ArcGIS Server periodically stopped

[Nadia_Matsiuk]

Hello!
I would like to ask for help.
Our team uses ArcGIS Enterprise 11.2 on Windows.
We are experiencing an issue where access to ArcGIS Server is lost periodically. Access can be restored by manually restarting the ArcGIS Server service.
For now, we are using a .bat script to restart the ArcGIS Server service every 10 minutes.

I haven’t been able to find tools to properly diagnose ArcGIS Server for errors. Error reports are not being generated, and I don’t see any errors in the logs that would help me identify the cause of the issue.

This problem started occurring after some changes were made. I believe it might have been a system update, but I’m not certain. The issue has been ongoing for about a month. Over the past few days, the service seems to be working more stably. We’ve even disabled the restart script and are monitoring to see if the situation repeats itself.

In general, this ArcGIS Server service includes Routing tools that we use in our projects. We recently updated the licenses for ArcGIS StreetMap Premium Europe, and the issue seemed to start around that time. There might also have been an update to ArcGIS Enterprise to version 11.2 (again, I’m not sure since this was done by another colleague). It seems that after installing the new license, the problem began. I can confirm that this happened before we published the RoutingTools.

Could anyone suggest methods or tools to identify and resolve the issue? Or has anyone encountered a similar problem before?

I would be grateful for any help!🙏

[OzanCanT]

Hello, 
It sounds like a frustrating situation, especially when the root cause isn’t immediately obvious. Let me try to help by asking a few questions and offering some ideas that might guide you toward a solution:

• When you mention "access to ArcGIS Server is lost," does this mean that the Windows service itself is stopping, or is it just that users cannot access the server (e.g. through the Manager endpoint)
• Have you checked the Event Viewer on the Windows Server hosting ArcGIS Server? You can look under the Application and System logs for any warnings, errors, or changes occurring around the time the service becomes inaccessible. This information might help you identify the problem.
• In the ArcGIS Server Manager, have you tried lowering the log level to Debug or Verbose under Logs > Settings? This can provide more detailed information that might help pinpoint the root cause.
• In the ArcGIS Server Manager under Site > Software Authorization, does it show that the licenses are active and valid?

Let me know if any of this helps or if you find something interesting in the event viewer or ArcGIS Server logs. Feel free to share more details

[Nadia_Matsiuk]

@OzanCanT , hello!

Thank you very much for your response!💛

Here’s some additional information:

1. Regarding "access to ArcGIS Server is lost":
   The ArcGIS Server service in Windows Services itself appears as running. However, when I try to access .../arcgis/rest/services/Routing/NetworkAnalysis/NAServer, I receive an error 500. I’m also unable to access ArcGIS Manager.

2. Thank you for suggesting the Event Viewer! I would like to learn how to use it effectively.
   Here is the error I see during the time period when the issue is reported in ArcGIS Manager:

   01/14/2025 20:45:04
   Error
   The machine-default permission settings do not grant Local Activation permission for the COM Server application with CLSID {…} and APPID {…} to the user <domain name>\GISadmin SID (…..) from address LocalHost (Using LRPC) running in the application container Unavailable SID (Unavailable). This security permission can be modified using the Component Services administrative tool.

   01/14/2025 20:45:12
   Information
   The ArcGIS Server service has entered the stopped state.

   01/14/2025 20:45:26
   The Network Setup Service has entered the stopped state.

   01/14/2025 20:47:35
   The Windows Modules Installer service has entered the stopped state.

   ...

   01/14/2025 20:55:02
   The ArcGIS Server service has entered the running state.

   Afterward, there are about 20 similar errors (The machine-default.....).
   Then again, ArcGIS Server service stopped → running.
3. Yes, I’m trying to find answers in ArcGIS Manager > Logs. Here are some logs that I believe are returned when a routing request fails. I’ve noticed that this chain of errors has been repeated frequently. But I don't see any error reports.
   
   01/14/2025, 20:45:09
   Error executing solve closest facility Error handling service request :Service not started.
   01/14/2025, 20:45:08
   Service containing process crashed for 'Routing/NetworkAnalysis.MapServer'. Please see if an error report was generated in 'C:\arcgisserver\logs\<domain name>\errorreports'. To send an error report to Esri, compose an e-mail to ArcGISErrorReport@esri.com and attach the error report file.
   01/14/2025, 20:45:01
   Instance of the service 'Routing/NetworkAnalysis.MapServer' crashed. Please see if an error report was generated in 'C:\arcgisserver\logs\<domain name>\errorreports'. To send an error report to Esri, compose an e-mail to ArcGISErrorReport@esri.com and attach the error report file.
4. The corresponding licenses are activated and valid in Software Authorization. Everything is fine on that front.
   It might be worth mentioning that, overall, everything works without issues when it comes to generating routes. The problem is specifically with ArcGIS Server “crashing” periodically and requiring a restart.

To reiterate, for the past few days, ArcGIS Server service has been functioning stably without such errors. I have disabled the automatic restart script (which was set to run every 10 minutes). So far today, the server has been running successfully without issues. I am therefore reviewing logs from the period when I know these errors occurred. I'm afraid the situation will happen again, so I'm trying to investigate the reasons.

Thank you once again for your help! It means a lot to us. For the most part, when such issues arise, the Esri Community is the only place where we can get advice from someone experienced. Thank you so much!💛

[OzanCanT]

Thank you too for your contribution to the Esri Community 😊. This is a very useful place to find solutions for users with similar issues. 

It sounds like the ArcGIS Server service isn’t stopping entirely but rather running into errors that prevent it from properly handling requests, particularly for your Routing service. Here’s some additional information that might help clarify the situation and guide your troubleshooting efforts if the crash occurs again.

Since the errors specifically mention crashes of the Routing/NetworkAnalysis.MapServer service, I’d recommend checking the following path to see if any additional details are captured about why the service is crashing.

• \arcgisserver\logs\MachineName>\services\<YourServiceName>
• \arcgisserver\logs\<MachineName>\errorreports

Since this is a routing service, it might be under heavy usage, which could lead to resource exhaustion. To check:

• Open ArcGIS Server Manager and go to Services > Routing > NetworkAnalysis > Pooling.
  Confirm whether the service is configured as Dedicated or Shared.
• If it’s Dedicated, check how many ArcSOC processes (instances) are configured. A high number of instances can strain the server’s CPU and RAM if the hardware can’t handle it.

Since the service is described as “crashing” frequently, it’s important to check for system resource bottlenecks. When the issue occurs, is CPU or RAM usage spiking on the ArcGIS Server machine? Tools like Task Manager or Resource Monitor can help track resource usage over time.

About the logs in the Event Viewer, it may be necessary to make sure that the service running user has the required Local Activation permissions in the DCOM configuration. On the machine with ArcGIS Server installed, I recommend running the Configure ArcGIS Server Account tool with the Windows user credentials running the ArcGIS Server service to ensure that there are no permissions gaps.

You mentioned that the problem may have started after some system or software updates. It might be worth reviewing the Windows Update History to identify updates installed around the time the issue began. Sometimes, specific updates can cause unexpected behavior with services.

Since everything works fine when generating routes, it might also help to monitor the frequency of service requests and their success/failure rates. In ArcGIS Server Manager, you can go to:
Logs>Statistics, and select your Routing service to view usage patterns and total request size. This might reveal whether high usage correlates with the crashes.

I hope ArcGIS Server and the Routing service will continue to run smoothly!

[TimWestern]

One thing I'll add is if this is Local Enterprise Server, also make sure disk space hasn't been used up.  I recall one time arcgis became unresponsive, and something else on the server had gotten in a state where logs filled up remaining available drive space.  

[Eugene_Adkins]

Could be unrelated but this post from a couple weeks ago sounds like it deals with a similar issue after an update to version 11.3. The post is not marked as solved but note @RandyBonds_Jr_’s last reply where a few steps seemed to resolve the issue for him.

Hope that helps.