Suddenly we are having trouble with our Relational Datastore on ArcGIS Enterprise 10.5..
Looking in the ArcGIS Server Manager logs we can see messages such as :
Service containing process crashed for 'MyService.MapServer'. Please see if an error report was generated in 'C:\arcgisserver\logs\SERVERNAME\errorreports'. To send an error report to Esri, compose an e-mail to ArcGISErrorReport@esri.com and attach the error report file.
Any ideas? Could it be a hosted layer in the DataStore which is causing the problem?
Any help would be great.
I ran into this issue after updating my licensing for ArcGIS Server. Running the ArcGIS Data Store updatelicense command resolved this issue:
1. Open command prompt as an Administrator on the ArcGIS Data Store server
2. Set the work directory to <ArcGIS Data Store installation directory>\datastore\tools. Ex:
cd C:\Program Files\ArcGIS\datastore\tools
3. Then simply type updatelicense to run this command
my licence has not changed but I ran updatelicense anyway and I get the message "Update license failed for managed geodatabase 'db_71scs'".
I can't understand how the DataStore appears valid in the admin page (see below) but cannot be validated in ArcGIS Server Manager or Portal. Any ideas?
There are two ways to validate the ArcGIS Data Store, one is Server validates it, (validating the Data Store through Manager or the Admin API), and the other is Data Store validating itself, (which is what you've done to receive the response you did). I'm not exactly sure what Data Store is checking when it's validating itself, but the "trust" between Server and Data Store seems to be broken somehow. It's strange that it works for a while before failing. Are there more details about why it failed when you look at the Server logs?
Hi Cody, overnight the datastore seems to have sorted itself out and now appears valid in ArcGIS Server Manager. I'm still testing to see if everything is working but I'd like to establish what might have happened. ArcGIS Server (also the Hosting Server) and the Relational DataStore are both on the same machine. Is this causing problems? Would it be a resource issue?
I am glad to hear that it is now working. More than likely this could either be a network issue(since it just started working) or a resource issue as you stated. ArcGIS Server and Data Store being on the same machine is normally not an issue but if they are ever competing for resources then that would explain the behavior we are seeing.
It is not uncommon to see behavior like this if for some reason the machine was being pushed updates or maintenance was doing done on the Server unbeknownst to you. I would say monitor the machine for a few more days to see if the issue comes back, or speak with your System Admins to see if there was anything pushed out during the down time.
Might be entirely unrelated but thought I'd share the unusual behavior I've seen which is strikingly similar to yours.
Running a full stack on a single instance (portal/WA/server/DS/DB). All would be good one second, and the next everything would be in tatters. Server/Portal would turn unresponsive or start getting cert errors - everything just tanked. Try to sign-in with Pro and it sometimes gets in an endless loop with Portal or simply crash dump.
Narrowed down part of the cause to the "Internet Connection Sharing (ICS)" Windows Service. This ICS was changing the FQDN suffix of the machine from something like "Machine.MYDOMAIN.COM" to "Machine.MSHOME.NET". The enterprise components would pick it up and reconfigure - Data Store squirrels the bogus FQDN name off into one of its Postgres databases to revisit at another inopportune time, oh yea - and it backs it up!
Googling "turn off ICS" will lead you to an array of sites that direct you to shut off network sharing on each network connection, but that's no good. Others will suggest stopping and setting the ICS service to "Disabled". Nope - it comes back alive.
The real root cause is Windows Update Service - which conveniently enables a disabled ICS service, then starts it, causing a cascading series of errors - none of which play out exactly the same way twice, just to keep you guessing. I finally disabled the blasted thing (windows update) using the group policy editor - then things started stabilizing once I rebooted, reinstalled web adaptors and made sure no trace of the rogue machine FQDN
The moral of the story is ... if it seems to happen at "freak" times - check and see if coincidentally you just so happen to have some new windows updates ready to install? You might.