Data store fails validation

[RobStauder]

Hi all,

I'm looking for some suggestions to help me resurrect our ArcGIS Enterprise. Here's what happened - 

1. Errors occurred while publishing some very large scene layers.
2. The output services did not start, so services were manually restarted and perhaps the entire site was restarted. This also caused errors due to an expired certificate.
3. The expired certificates were replaced with self-signed certificates.

However, now:

1. The site will not validate through the settings section in the portal
2. The relational data store will not validate in the server manager (but the tile cache does). 
3. The data store does validate through the server's rest validate operation (../admin/data/items/enterpriseDatabases/AGSDataStore_ds_blah/server_blah/validate)
4. Some of the services will not shutdown or restart. Eg - the system SceneCachingTools service will not start, and throws error java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: java.net.ConnectException: Connection refused: connect, while our network routing service (map server) does not respond, won't render in the map and will not restart.
5. We cannot publish new feature layers or scene layers, but we can view scene layers and feature layers published before the crash. 
6. describedatastore.bat returns a status of started for both relational and tile cache. Mode = readwrite
7. The windows application log has a few errors
   1. Notification of policy change from LSA/SAM has been retried and failed. Error 4312 to save policy change for account

   2. The description for Event ID 0 from source PostgreSQL cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer. The following information was included with the event: Is server running?

Any ideas? Should I try updatesslcertificate.bat with the self-signed certificate (as this is the only certificate I can't view)? 

Thanks!

Rob

[GeoJosh]

Rob, anything of interest in Data Store logs? C:\arcgisdatastore\logs\<machineName>\server

Did you apply a new license to ArcGIS Server by any chance?

[RobStauder]

Thanks for the response. The only thing in the logs is Validate: Failed to check layers extent. No new licensing. I just checked the software authorization in the server manager and we are good through 11/22.

[Ikebana]

Hello, did you only change cert for the datastore?

[RobStauder]

Hi. Thanks for the response. That's the only one I can't confirm was changed. I know that the ssl certificates for the portal and server were updated to use the self-signed certificates. I don't know how to view the data store's certificate, so I'm not sure if it was updated. 

[Ikebana]

Sounds to me like a Cert issue. But I have also experince that only tile cache datastore worked. My solution was to uninstall datastore --> reinstall --> import backup.

Is it a single machine deployment?

[HenryLindemann]

Hi @RobStauder,

You can check you datastore cert  by opening the url https://localhost:2443/arcgis/datastore then looking at the cert info.

You have two problems your ArcGIS Server is not functioning correctly and your Datastore is not validating.

First make a backup of the system

Then can you go to services and look which service account is running your ArcGIS server then reapply the security rights on the folder structure at c:\program files\arcgis\server and the location of you configstore c:\arcgisserver or d:\arcgisserver...

Service account

config-store

Program Files

right click and open server properties then security and your service account make sure it has full control on these folders, there is a new Windows policy that changes the service account to read only. 

after you have applied this you can check if your ArcGIS server is functioning.

Then there might be a partial file that has been created during your failed publishing go to  C:\arcgis\arcgisserver\config-store\services\Hosted and see if you can see anything also your rest and services will not work if this is the case 

https://dns:6443/arcgis/rest

Datastore
if you can make a backup then @Ikebana suggestion will work and do a nice clean rebuild of datastore but your ArcGIS server must be in a working state.

You can also try the following go to server manager and unregister the datastore then re-register it on https://dns:2443/arcgis/datastore

you only have to unregister relational NB did you make the backups

using the https://dns:6443 admin url

if it still fails then go to the tool 

"C:\Program Files\ArcGIS\DataStore\tools\unregisterdatastore.bat"

and try the 2443 url again

If it still fails then the only option left is to restore the backup

to create the backups run
"C:\Program Files\ArcGIS\DataStore\tools\backupdatastore.bat" --store relational

and 

"C:\Program Files\ArcGIS\DataStore\tools\backupdatastore.bat" --store tileCache

please make sure that  they did actual make a backup in the backup location

and 

something like this for relational

to restore uninstall ArcGIS datastore and remove all the files except backup in c:\arcgisdatastore or your location
 

relational then edit and run these commands
"C:\Program Files\ArcGIS\DataStore\tools\restoredatastore.bat" --store relational --target most-recent --source-loc D:\arcgisdatastore\backup\relational --data-dir D:\arcgisdatastore --server-url https://dns:6443/arcgis --server-admin adminuser --server-password adminpassword --loaddata true --prompt no

tileCache
"C:\Program Files\ArcGIS\DataStore\tools\restoredatastore.bat" --store tileCache --target most-recent --source-loc D:\arcgisdatastore\backup\tilecache --data-dir D:\arcgisdatastore --server-url https://dns:6443/arcgis --server-admin adminuser --server-password adminpassword --loaddata true --prompt no

don't copy and past the above string type it out please you might get encoding problems

Regards

Henry

[RobStauder · Accepted solution]

Thanks to everyone who replied. I resolved it. I followed the suggestions in this document - 

Problem: Unable to start or stop services in ArcGIS for Server (esri.com)

I found the word undeploy in one of the logs, which led me to that article. 

Once the system was back up and running, and I was able to publish feature and scene layers to the portal, I had a conversation with someone from Esri tech support. The analyst confirmed that the certificates and permissions were fine. 

I appreciate all the help from this forum and Esri tech support. 

Thanks all!

Rob