ArcGIS Enterprise/Server - ArcSOC not running

[AbiDhakal]

Hello GIS friends,

I hope you are doing well. I am currently experiencing an issue with our GIS Server. Each time I restart the GIS Server machine, not all of the ArcSOCs start automatically. This has become a significant problem, as it is preventing our mapping applications from opening.

As a temporary workaround, I have been manually going to the Services panel to restart ArcGIS Server, ensuring all the ArcSOCs run properly, and the mapping applications function as expected.

Any assistance or guidance you can provide would be greatly appreciated.

Thank you in advance for your help.

Best regards,
Abi

[JM32]

@AbiDhakal 

I don't know too much about the SOCs, but from my understanding if you are running shared instances on services - the SOCs will be turned off until someone starts using the layer. Dedicated instances will always have their SOCs running 24/7.

You can check this in server manager and change the settings in there for the layers.

Cheers,

Jon

[RyanUthoff]

Dedicated instances don't necessarily mean they will always be running 24/7. Dedicated instances will honor the min/max setting. So, if the min is set to 0, then they will not be running 24/7. Even if the min is set to 1 and the max is set to 5, only 1 will be running 24/7 and the other 4 will be running depending on the demand.

[TimoT]

Are there any errors in the server logs relating to instances failing to start or crashing?

When all your arcsocs are running, how many are running in total? Use the following technical article to count

• https://support.esri.com/en-us/knowledge-base/how-to-identify-the-number-of-arcsoc-processes-in-a-wi...

[AbiDhakal]

@TimoT  - 

Upon reviewing the server logs, I do not see any errors indicating that instances are failing to start or crashing. At present, I have 40 ArcSOCs running.

Here is a brief description of the situation: restarting the server now takes approximately 20 minutes, whereas it previously took about 10 minutes. This delay seems to have occurred after upgrading to Enterprise 11.4.

Let me walk you through the process as it unfolds:

1. I initiate the restart, and the machine begins the process.
2. For the first three minutes, everything appears normal. In the Task Manager, I can see six instances of java.exe running, along with javaw.exe for the server.
3. After this initial phase, however, the server stalls. Even the system time on the bottom right remains frozen.
4. Around 10 minutes later, I see the javaw.exe for the datastore start running. Three minutes after that, the javaw.exe for the portal begins to run.
5. Once this process completes, I attempt to open a map application. After some delay (approximately 5 minutes), the map services begin to load. However, I then receive an error message stating that some of the feature services could not be loaded.
6. At this point, I restart the ArcGIS Server from the Services panel, and everything begins working as expected.

In total, this process takes around 20 minutes.

Thank you very much for your attention to this issue. Please let me know if you need any further information or if there is anything else I can provide to help resolve this.

Best regards,
Abi

[RyanUthoff]

---

After this initial phase, however, the server stalls. Even the system time on the bottom right remains frozen.

---

This is concerning and would at least partially indicate this is a server resource problem and not an Esri problem. If applications outside of Esri, whether it be the system time, your entire RDP session, etc. are frozen when your server restarts, this would indicate there might not be enough server resources.

The other thing that isn't necessarily helping is that you have all three ArcGIS Enterprise components installed on the same machine. Keep in mind that all three components (along with anything outside of Esri) are all competing for the same server resources.

Perhaps one of the next things to check is when you restart your server, monitor your server resources (CPU/RAM) and see if they are maxing out within the first 15 minutes after a server restart. If they are constantly maxed out, then I think a server resource issue could be partially to blame.

Restarting a server is a pretty resource intensive process and I wouldn't be surprised if you're just maxing out resources. 

[abdelsalmelmoursi]

@AbiDhakal 

As i see you are using one machine for you enterprise as (ArcGis server , portal , Data Store,..)

you can check the following

1- your service account which used for arcgis server and portal service is same one and have full control on all the enterprise  directiores such as portal and server installation folder , python27 , Arcgis folder

2- check your cpu and memory utilizationit should be less than 60%

3- if you have antivirus try to disable it or try to whitelist your enterprise directories. (you can check event viewer for more info)

good luck

[kdoshiwork]

There might be a patch for ArcGIS Server for the version you are using.

Applying that might resolve your issue.