ArcSOC.exe process is running at high cpu usage

[ShaunConway]

ArcGIS Server 10 .Net
Windows Server 2008R2 Standard
RAM 4GB
Intel Xeon 2.67GHz (2 processors)

We have noticed that there is a persistent ArcSOC.exe process running between 20-30% CPU usage on our windows 2008R2 machine. We have stopped all services on the server and believe that this remaining process is either dedicated to logging or directory management (based on esri documentation.) If the process is stopped, it immediately regenerates.

What we are trying to figure out is why this process is running at such a high CPU usage. This is our development machine, and the identical production machine is not experiencing the same high CPU usage. The settings for ArcGIS Server are identical. In testing, I set the logging to verbose expecting to see an increase in the CPU usage associated with this process, but none occurred. I am not sure what "directory management" is and am therefor struggling to test whether or not this is the cause.

Any suggestions or help would be greatly appreciated.

[ShaunConway]

Hey Rex!

Hope all is going well. It's been a while, so I don't remember exactly what we did to resolve this, if anything. To be honest, we may have just stuck it out until an AGS upgrade, but our issue was of course on development which allowed us that flexibility. There were are few issues we had going on around this time, so here are some things we did around that time to resolve issues ... maybe one of them will be useful?

 - Created a new, identical map service and used that in place of the one with the issue

 - Restarted AGS and then the server itself (I know this is an obvious one, but it's amazing how many times it has resolved our issues.)

 - Republished the service, removing one layer at a time, to determine if one layer was the culprit or if it was an issue with the service/AGS.

 - Republished the service pointing to a copy of the data. Copy of the data wasn't a copy/paste or import, but a clean feature class, import schema, then append.

Also I'll double down on David's suggestion of Anti-Virus software. I know that was a major issue for us at one time in a previous AGS release. Simple rule change cleared it all out.

Sorry I couldn't be of more help!

Shaun

[RexRobichaux3]

Thanks again Shaun and David for the replies and help! I have a few updates to pass along just in case anyone suffers this CPU behavior going forward. It looks like there are a myriad of issues causing our problems for our production server. The steps we have taken and the issues that have been identified:

• We migrated our VM server to a new host with better processing resources. Our production server seemed to like this as even though we still had higher-than-we-used-to-have levels of CPU utilization, it was no longer maxing out resources and bringing apps and services to a haulty- so we likely had some underlying hardware issues.
• We further improved performance by adding all of the arcgis server directories to the exemption list of (both) antivirus applications running on the server. These same antiv applications run on all of our servers including dev, but they seemed to be going a little wild on production- IT is looking into why. With all of this done CicsoAMP still seems to be going crazy so we are looking into why CiscoAMP is having issues in this environment. We have had to completely deactivate CiscoAMP for the time being until we figure out why. The directories we admittedly over-zealously added: 
  • c:\arcgisserver
  • c:\Program Files\ArcGIS\Server
  • c:\Python27
  • c:\Program Files\Esri
  • c:\ Program Files\Common Files\ArcGIS
  • c:\app\Oracle (our oracle client on the server)
  • c:\Webservices\Data (our file geodatabase storage directory for GIS published data)
  • e:\arcgiscache (cached aerials directory on the server)
• It appears that using the search widget in any app, without specifying a map service layer, is a major contributor to the CPU usage. As any one of our apps has between 15 and 25 searchable layers, this is not really surprising. Therefore we will have to continue with more end-user education that when using the search widget, the layer of interest needs to be specified. We have always been pretty religious with this messaging but there maybe a new hire or app user who doesn't know this or has forgotten that is unknowingly really slowing things down. I also have a case open with Esri support to verify that this issue is expected behavior and not a bug. 

Hope this helps guys- if we figure out anything else I'll update this.  Thanks again for the info and assistance!

-Rex

[LakshmananVenkatesan]

HI Shaun,

We are also facing similar kind of issue, did you got any information or updates on this problem