This past weekend, we attempted to upgrade our ArcGIS Enterprise to 11.5 from 11.3. However, we ran into issues with the datastores and ultimately had to roll back to 11.3. I wanted to see if anyone else has run into similar issues, here are some details:
1) No issues upgrading Portal or the Hosting Servers to 11.5
2) When upgrading our Datastore which has relational and tile cache (we will be moving to object store on separate servers), the upgrade ran fine, but when we went to configure the datastore. We received an error on the configure webpage that it could not reach /datastoreadmin/config?f=json. Also while tried to use the command line, we received an error that it could not reach localhost.
3) Received the same error when upgrading Spatiotemporal as well
4) Contacted Premium Support and they tried several options, including restoring to a backup, however, all the backups failed to restore.
5) With this being a production system, we decided the best option was to roll back to avoid any additional downtime.
Some details about our environment:
1) All servers are Linux, RHEL 8
2) Portal, Hosting Server, and Datastore are all HA (two machines each)
3) The upgrade was first done on the primary, then the standby Datastore, after which the configure was attempted on the primary (secondary also showed the same errors)
4) Something that was noticed is that the queuestore_config.json file in the /etc on the relational/tile cache was still referencing 11.1 and showed that its last known state was Upgrading.
Our thoughts for next streps are the following:
1) Try the upgrade again and see if it works this time.
2) Try moving the datastore to two new servers
a) Provision two new servers
b) Install Datastore 11.3 on one of the servers, 11.5 on the other
c) Add the new 11.3 datastore server to the cluster
d) Promote the new 11.3 datastore to be primary
e) After it has been verified that content has been replicated over to new server, remove the two old servers
f) Upgrade the new 11.3 datastore to 11.5 and configure
g) If configure doesn't work, roll back to 11.3 snapshots. If it does work, add the new 11.5 datastore server to the cluster.
3) Create a new backup and try restoring again.
4) Download all hosted services as file geodatabases and republish everything. There are maybe 15-20 hosted services in our environment.
If anyone else has run into similar issues, I'd really like to hear how you went about resolving. Again we do have a premium support case and may even see if we can get professional services to assist.
