Hello Erin,
I feel your pain! I have actually had alot of issues with the locator in ArcGIS Pro as well. Due to the fact that ArcGIS Pro is based on "projects" and anything you save in those projects is preserved for THAT project. There are ways to make some settings stick like folder, database connections, etc., but the locator is not one of them unfortunately! Now... for a way to remove locators you don't need. The world geocoding service is default to the locator as is the x,y locator. As you mentioned, you can turned them off in the provider settings but somehow they get turned back on in a new project or other instances of the same project. UGH!
When you set up your search layers... are they layers that are in your project or stored elsewhere? This may be causing some of the issue because you said you have to sign in or out of AGOL to resolve the issue?
Another idea is that you could create your own locator and then add it to your ArcGIS Pro provider settings. I did this for our office and as long as the analysts use the same project where the locator was added, they have it available to them. Of course there are caveats to this solution where you must figure out how to update the data behind your locator because data like addresses or parcels change often. If you need additional help with this, I would be glad to share my workflow process with you.
Amanda Bishop, GISP