Currently, when you have the Symbology pane open at all times, and click on a layer symbol in the Contents pane for the first time so that the Symbol Gallery has to load, you can see how Pro loads the various styles into memory (the changing/shrinking scroll bar of the pane is a give-away) and it takes a little while before you can interact with the symbol of your liking.
When you click on other layers to do symbology things soon after, the gallery will then show up instantaneously.
If you leave symbology alone for a while and then request it in order to pick something else from gallery, it will be slow and appear to load all the symbols from disk again.
It appears that Pro offloads unused(?) items from memory.
For those of us with serious hardware it would be a nice option to be able to set items such as the symbol gallery to remain in memory at all times so that the overall Pro experience is more efficient.
The same issue can be seen from many other Pro components but the Symbology Gallery is an often used aspect of Pro.