ArcGIS Enterprise 10.8.1 HA: Make Datastore Machine Primary failed

Question

Hello,

I am trying to promote my datastore standby machine to primary for the tileCache datastore but it fails with the following error:

Server machine 'https://Machine1.domain.com:2443/arcgis/datastoreadmin/machines/Machine2.domain.com/makePrimary' returned an error. 'Attempt to make 'Machine2.domain.com' the primary data store machine with role 'PRIMARY' is not allowed.'

When I try with the relational datastore, it works.

Validating datastores on Admin REST API, returns both machines as healthy.

Any idea ?

Thanks,

Best regards,

Nicolas

ShanonLoughton · Answer

Hi
You're a few versions ahead the OP's problem, so almost warrants a new ticket. Some troubleshooting steps:

Basic functionality and healthy deployment:
1.a First verify the two tilecache machines validate and are healthy. Also if no published scenes it would be a good test to ensure that you publish one and ensure at least basically working as seen from Portal and/or ArcGIS Pro.
b. Check Datastore Tilecache logs to see any more detail on the issue(s)?

If startup timing issue like mine:
2. Reset the machines or Windows services on given machines, wait at least a few (3-5) minutes, then try. This includes restarting ArcGIS Server.

Or if other permanent issue:
3. a. Check ports open between ArcGIS Server <> Tilecache Datastores (HTTPS 6443, 2443, 9876(?), 29081, HTTP 29080), as well as between Tilecache Datastores too (TCP: 4369 and 29085-29090). Obviously that is in the infra networking as well as Windows firewalling. https://enterprise.arcgis.com/en/system-requirements/latest/windows/pdf/ports-enterprise-deploy-dgm.pdf

b. Clear ArcGIS Server's REST cache might fix up a funny map/fs/gp/tile scene state that is blocking the makePrimary command. I do not know for sure but it might also have some other information about the current state of the HA of datastores, so it might refresh that too. https://developers.arcgis.com/rest/services-reference/enterprise/clear-rest-cache.htm

c. Each underlying postgres DB on datastores needs to be accessible from the other machine's IP or hostname. Insecurely opening that up to allow all temporarily and trying makePrimary might work. pg_hba.conf is in the datastore configuration directory somewhere (sorry not sure exactly right now). https://stackoverflow.com/questions/3278379/how-to-configure-postgresql-to-accept-all-incoming-connections

More drastic - reinstall / re-join / reconfigure the standby:
4. If the standby is determined to be the problem then there are few ways to reinstall it/wipe its config and restart service and reconfigure, and then try makePrimary after that, similar to this:
https://support.esri.com/en-us/bug/changing-the-tile-cache-data-store-mode-in-arcgis-enter-bug-000150950

hth

Olivit · Answer

Hello,

it's been a while, same identical problem here with ArcGIS Datastore 11.1 on Windows Server 2022 Datacenter.

Using hosting server admin rest interface and promoting relational datastore standby machine to PRIMARY worked very well (together the sidecar queue store), while trying to promote the tileCache datastore keep generate the very same error.

Any hint really appreciated!

Thanks and regards,

Oliviero

ShanonLoughton · Answer

Ive hit this recently on newly deployed MM HA 10.8.1 with a handful of small published scene layers / existing data.
I had restarted the whole machine cold tile cache data store Machine1/current PRIMARY, and then did as  you did trying to make Machine2/current SECONDARY > makePrimary command. Whilst after a first boot, or similar like starting the ArcGISDatastore.exe service or clicking Start in admin UI, on Machine1/current PRIMARY it can take some time before it listens / allows actions.
Waiting another 5-10 mins and then doing Machine2/current SECONDARY > makePrimary action worked for me.