ArcSOCs Exceed Max Instance Setting

[MattFancher1]

Has anyone ever had a problem where the number of ArcSOC processes running per service exceed the max number of instances set for the service. For example, I have the max number of instances for a map service set at four, but when I look in task manager, I may see there are 10 ArcSOC processes associated with the service. When that sort of growth occurs across the 20 or so services I have published, the server eventually runs out of memory and services become unavailable.

Oddly, this has started happening on two production servers almost simultaneously. The servers ran with little to no issue all of 2017, but we've had 14 incidents spread across these two servers since the end of January. To make things more fun, the servers are somewhat different. Server #1 is Windows 2008 running ArcGIS Server 10.2.2. Server #2 is Windows 2012 running ArcGIS Server 10.3.1. Both are single machine sites. We run all services in high isolation.

I've tried a number of things on both servers:

1. Adding additional ArcGIS-specific folder exclusions our anti-virus

2. Making ArcSOC a trusted process in anti-virus

3. Making pkill a trusted process in anti-virus

4. Running the configure ArcGIS Server account utility

No help there.

Some other odd things:

1. I can't run pkill manually on either server unless I add the ArcGIS Server account to the admin group

2. Ending the extra ArcSOC processes from task manager has a cascading effect. If I end one, 10 more could die off

For the record, I have engaged Esri support on this issue. Progress has just been slow, so I wanted to put this out there in case is resonates with anyone.

Thanks in advance for any info!

[DavidColey]

Matt - from what I understand if you only have the mapping capability setup on a service, it is not possible for the service to exceed the maximum number of socs you have set.   But if a service also has feature access capability, then you could have 4 soc instances for the mapping capability and 4 soc instances for the feature access capability, or 8 socs.  That could vary more depending on what capabilities you have set for a service.

[MattFancher1]

Hi David. Thanks for the reply! We do have a few services with feature access enabled, but most just have mapping and maybe WMS enabled. But even if enabling WMS doubles the number of ArcSOCs created, I don't think the math works out. For example, I have a map service with WMS enabled and max instances set at two. If enabling WMS doubled the number of ArcSOCs created, I'd be at a maximum of four. But during the last incident on the server there were 12 ArcSOCs for the service. Restarting the ArcGIS Service service brought the number of ArcSOCs back in line.

[DavidColey]

In that case sometimes the soc instances become hung, that happened a lot on the older ArcServer versions, especially after Windows security updates - at least for us.  Hard to say for you since your running in a mixed environment.  Hasn't happened to us since we moved to 10.4.x on Windows Server 2008 r2 (production) or at 10.5.1 on Windows Server 2012 (dev).  

[MattFancher1]

Yeah we suspected a Windows patch or security update could be the issue, and it may well be. However we started having problems more than a month after our previous round of patches, and these servers see a moderate to heavy daily load.

[AnibalMmartinez]

Matt, ¿no tienes ningún ERROR en el log de ArcGis server?

[MatthewLofgren]

I have experienced this as well. 

Windows 2012R2 

ArcGIS 10.4.1

Multiple times have I had low memory warnings only to find a lot of instances of service in task manager. I shut the service off and a few of the ARCSOCS (tied to that service) go away, but many are left behind. I just end the processes in task manager, turn the service back on and schedule a reboot for that night.

I first noticed this this past December. It has happened at least 5 or 6 times since.

[MattFancher1]

Since my first post I found that ArcGIS Server seems to lose track of some ArcSOC processes. I periodically check by running the following command:

wmic process where name="ArcSOC.exe" get name,processid,creationdate

That produces a list of running ArcSOC processes with the date/time they started. For me running that typically finds a couple ArcSOCs from previous days. All our services use the default setting to recycle at midnight, so I really shouldn't have any ArcSOCs from previous days. If I use task manager to kill the old ones, then the server stays stable. If the server gets into a low memory condition, I've found killing off the old processes produces the "cascading effect" I mentioned in my first post. It's not a good solution, but it's keeping our servers from crashing until we can either figure out the root cause or rebuild the servers.

[MatthewLofgren]

Cool. You can also add Commandline to that command to get info on which service the ArcSOC is tied to.

wmic process where name="ArcSOC.exe" get name,processid,creationdate,Commandline

[RandallWilliams]

Keep in mind that when you set max instances, it's max instances per ArcGIS Server machine in your site. So if you have a 3 machine site, and a service is set to 4 instances, it's possible to have 12 total instances running for that service. This doesn't take into account clustering (where you could assign service workers to specific machine instances), which is deprecated in ArcGIS Server