All Web Services Shown as "Stopping" in ArcGIS Server 10.6.1

[Clint_Glover]

I am currently running three environments (development, acceptance, production) of ArcGIS Enterprise 10.6.1 on Windows 2012 servers (Portal and Server are federated) and I am experiencing an issue (only on the acceptance environment) where all of our web services within ArcGIS Server are shown with a status of "Stopping". 

In the ArcGIS Server logs, two big error codes appear after each attempt to restart the ArcGIS Server windows service.  The error codes are 6550 (failed to start the server machine) and 7365 (error while starting AppServer).  I began my troubleshooting by restarting the ArcGIS Server windows service.  After that didn't fix the issue, I rebooted the entire server (didn't fix the issue).  After that, I went through and checked multiple things including verifying that the windows account running ArcGIS Server has the correct folder permissions set, verifying that I have a valid licenses to run ArcGIS Server, verifying that the acceptance environment is configured the same as the development and production environments, and verifying that the acceptance environment has the same ArcGIS Enterprise patches installed as the other environments. 

I will note that this behavior occurred after some recent windows patches were installed on that server.  When comparing the timestamp of the errors appearing in the ArcGIS Server logs to the timestamp of the installation of the windows patches on the acceptance server, they are very similar (same day).  This leads me to believe that the recent windows patches had some affect on ArcGIS Server.  The odd thing is that the same windows patches were installed on the development and production environments with no issues arising after installation.  I contacted ESRI support and all they can tell me to do is to attempt to uninstall the windows patches to see if it corrects the issues or uninstall/reinstall ArcGIS Server in that environment.  I would rather not have to uninstall/reinstall ArcGIS Server due to possible issues that might arise from our federation with Portal.  This is the first time I have ever experienced this behavior in any version of ArcGIS Enterprise.  Does anyone have any other suggestions for things I can try? 

[JoshuaPust]

I had something similar to this where LAN-storage for my shared config-store was offline. I don't have a clue if your config-store is on the same machine with your server install.

[Clint_Glover]

The config store is located on the same machine and the same drive (D) as ArcGIS Server.

[jill_es]

Hi Clint - if you go through the workflow outlined in this Technical Article, are your services still stuck in a "Stopping..." state?

[Clint_Glover]

That was one of the first workflows I tried and it did not work.  I have tried it a few times on that server and nothing changes.  I will also mention when I log into the ArcGIS Server Administrator and go to machines --> machineName --> status I see that the configured state is set to STOPPED and the real time state is set to STARTED.  I went back to the machine page and clicked on the start operation.  It ran for several minutes and nothing happened so I stopped it.  I then went back to the machine page and clicked on the stop operation and then tried the start operation again.  This time it ran and both the configured state and the real time state were set to STARTED.  But when I went to ArcGIS Server Manager and tried to load the services page, it never loaded and none of the services were working (tried accessing web apps using those services).  Not sure if this info matters, but I figured it wouldn't hurt to share.

[Clint_Glover]

Jill, do you think that possibly running the repair option for ArcGIS Server might fix the issue?  Do you know if the repair will interfere with the federation?

[jill_es]

Hi Clint - thanks for confirming that you went through that workflow.  You could try a repair - it won't break the federation, but I've never seen it fix the "Stopping..." behavior you are encountering.

At this point, I would recommend reaching out to Tech Support to look further into this issue.  I'll be reaching out to you directly regarding this.

[Clint_Glover]

I have already reached out to tech support and a case was opened on 12/2/19.  I have been working with a guy named Steven.  We went through several workflows (including the one you mentioned above) and when none of them worked, his only other recommendation was to uninstall the recent windows patches and/or uninstall/reinstall ArcGIS Server.

[Clint_Glover · Accepted solution]

After working with ESRI tech support for several weeks, we finally found a solution.  It appears that one of our custom geoprocessing services (geometric network trace) was causing the issue with all of the services showing as Stopping.  We believe the service became corrupted some how.  Below are the general steps I followed to fix the issue (if you need more details let me know):

1. - Backed up ArcGIS Server directories (default locations - C:\arcgisserver & C:\Program Files\ArcGIS\Server)
2. - Exported ArcGIS Server Site (single machine site) via ArcGIS Admin page.
3. - Uninstalled ArcGIS Server & renamed the Server folder located at the default location of C:\Program Files\ArcGIS
4. - Reinstalled ArcGIS Server with the exact same configuration as previously used (had to create new empty Server folder at C:\Program Files\ArcGIS during installation)
5. - Created new site within ArcGIS Server with the same PSA info as previously used.
6. - Imported previous site via ArcGIS Admin page.
7. - After my site had been imported successfully, I had to stop the ArcGIS Server service and copy over the old security-config.json, which contained federation information, to the new install of ArcGIS Server (default location - C:\arcgisserver\config-store\security).
8. - Restarted ArcGIS Server service and verified via ArcGIS Server Manager that all of my services were shown as Started and ArcGIS Server security was set to Portal for ArcGIS (under the Security tab of ArcGIS Server Manager)
9. - Opened new tab and went to the associated Portal --> Organization --> Edit Settings --> Servers and validated the associated ArcGIS Server site (green check mark next to server name)
10. - Searched for and located the custom geoprocessing service within Portal and deleted the item.  Just to double check, I went back to ArcGIS Server Manager to make sure the service had been removed.
11. - Opened mxd file and run the associated model and published the result as a new service with the same name as the previously deleted geoprocessing service.
12. - After a successful service publish, I checked the service to make sure it was running correctly.
13. - To fully make sure the issue had been resolved, I stopped & restarted the ArcGIS Server service (all services came back up as Started) and then rebooted the entire server itself (all services came back up as Started).  After both of those checks, I deemed the issue resolved.

[kpatchell]

Hi Clint,

How where you able to identify which service was corrupt?

We have a similar situation , in 10.8.1, whereby the majority of our services are configured to be started but thier real state is stopped. Though restarting the server fixes the issue, it's not a viable long term solution as we use AWS Cloudformation to build Production servers every 2 weeks, so this will continue happening unless we find a fix.

Any help would be much appreciated,

Kind Regards,

Killian