Missing Hosted Feature Classes after upgrading to 11.1

[RobMcCulley]

We just upgraded our Enterprise installation from 11.0 to 11.1.  We followed this document to perform the upgrade.  Everything went well, and we had no errors or issues with the upgrade.  After the upgrade none of our hosted feature classes work.

The hosted feature classes still show up in Portal, but trying to access the data results in the message "There was an Error".  There is nothing in the portal log file regarding the hosted feature classes.  All of our hosted feature classes are still listed in the server rest services directory:

Trying to access any of the FeatureServer's results in an "not found" error.

There is nothing in the server logs related to the hosted feature class.

The federated server validates successfully in the Portal settings.

I have a daily backups of the datastore.  I tried restoring the backup created prior to the upgrade. The restore process was successful, but the hosted feature classes are still inaccessible.

We've tried restarting Datastore, Server, Portal, and restarting the machine, but the feature services are still inaccessible.

[JalesM]

Have you seen anything in the logs when attempting to access that data?

You could try unregistering the Relational Data Store from ArcGIS Server, clean uninstalling Data Store, reinstalling at 11.1, don't configure it, and then restore from the 11.0 backup.

Clean uninstall doc: https://support.esri.com/en-us/knowledge-base/how-to-perform-a-clean-uninstallation-of-arcgis-enterp...

Restore command: https://enterprise.arcgis.com/en/portal/latest/administer/windows/data-store-utility-reference.htm#E... 

[DavidColey]

In my experience this is a Server error, not a Datastore error or Portal error.  Something is preventing Server from communicating with datastore. For example, whenever something interrupts an overwrite of a hosted feature service via the arcpy sharing module, I will see this exact behavior. The hosted feature still appears in portal, the service still appears in the Hosted directory in Manager, and still shows in the Rest.  Yet accessing the layer in the Portal data tab will display 'There was an error'.

You say the federated site validates with portal, but does the relational datastore validate with server?

For Server Manager>Site>DataStores : if the relational datastore validates with server - have you been able to overwrite any of the services?  If you can, then you could just try overwriting the services.  If the overwrites don't work, can you write new layers to the datastore?  Time consuming of course.

If the relational datastore does not validate with Server I would try re-running the Server install. 

Regardless, this to me that says some type of communication loss happened during the Server upgrade and corrupted the configs. 

[ChrisUnderwood]

You could run a check on your Relational Data Store. That might help you to understand if the problem is with the Data Store itself, or somewhere else.

You can run a command like this.

"C:\Program Files\ArcGIS\DataStore\tools\describedatastore"

A healthy ArcGIS Enterprise 11.1 Data Store will return details like this.