ArcSOC.exe process is running at high cpu usage

[ShaunConway]

ArcGIS Server 10 .Net
Windows Server 2008R2 Standard
RAM 4GB
Intel Xeon 2.67GHz (2 processors)

We have noticed that there is a persistent ArcSOC.exe process running between 20-30% CPU usage on our windows 2008R2 machine. We have stopped all services on the server and believe that this remaining process is either dedicated to logging or directory management (based on esri documentation.) If the process is stopped, it immediately regenerates.

What we are trying to figure out is why this process is running at such a high CPU usage. This is our development machine, and the identical production machine is not experiencing the same high CPU usage. The settings for ArcGIS Server are identical. In testing, I set the logging to verbose expecting to see an increase in the CPU usage associated with this process, but none occurred. I am not sure what "directory management" is and am therefor struggling to test whether or not this is the cause.

Any suggestions or help would be greatly appreciated.

[nicogis]

Have you installed last service pack?

[ShaunConway]

Domenico,

Thank you for your response. We are running SP5 across the entire enterprise.

Thanks,
Shaun

[ShaunConway]

Here is a video of the ArcSOC.exe process. Note that all the other ArcSOC.exe processes are at 0%. This is our test server and these services are idle.


" rel="nofollow" target="_blank">http://www.youtube.com/watch?v=OwEUn7l13mQ&feature=youtu.be[/video]

[nicogis]

Have you details in event viewer Windows or log arcgis server?

[ShaunConway]

There isn't any critical information in the windows event viewer or AGS logs.

[RexRobichaux3]

Not to bring this back from the dead but- howdy Shaun!! Long time no see . This is a shot in the dark but why not: 

1) Did you guys ever figure out what was causing the CPU spike on the AGS for some SOC processes? 

2) If so, do you recall what the issue or culprit was?

We have just recently started seeing much higher than "normal" CPU utilization from 2-3 SOC processes (intermittent but reproducible with light service use) that just started in the last week. It's bringing our prod server to a near stand still and 100% utilization when there are several users hitting our apps from the services. Also nothing in the logs that jump out. Any help or ideas would be greatly appreciated- hope all is well in Lynchburg!

-Rex

[DEWright_CA]

What kind of services are you running that you see this issue with? Composite Geocoders can cause a larger load due to the nested data access.

Also various Geoprocessing tools can cause a spike after various OS patches to address security exploits from earlier this year.

Also consider taking a look at your Anti-Virus software; some configurations can see each time a service or its data is accessed that a AV scan of the data/files will be triggered which can cause the service to spin in a waiting-to-load state.

[RexRobichaux3]

Thanks for the reply David. We are seeing these delays / CPU utilization issues from standard map services that are referenced in a series of 10.3.1 Portal web applications. What is strange is that this was never an issue before a few weeks ago (checked and no Windows updates have been applied / nor Esri), and I cannot reproduce this behavior on our dev tier which mimics our prod tier. In the dev tier (and previously in prod) I can get CPU to spike to ~30% with some panning around (especially high multi-family areas where parcel drawing is more resource intensive) but never much more than that. The same app, same zoom extend and area panning now in prod will spike to 80-90% CPU and stay there for a little while after the draw is complete.

The only things that are different between prod and dev is:

• Our prod server (VM ware, 16GB RAM, 4 core hyper VM with Windows 2012R2, Intel Xeon E5-2695 V4) is a new "clone" of our old VM server as of a month ago. However IT has assured us that from a hardware perspective, our specs are identical. The dev machine has identical specs as well. 
• Our spatial data that is used for web app consumption come from two file geodatabases that also live on the respective servers. We recently switched from using one-way child to parent replication to update these file geodatabases (from our production enterprise geodatabase) bi-weekly, to a python script that truncates and appends all of the data from our Prod Oracle EGDB. 
  • Im currently in the process of making new copies of these FGDBs and deleting the GP history / metadata that has accumulated due to this recurring truncate and append. Hoping to test performance from the new FGDBs tonight.
• We also compacted these two FGDBs a few weeks ago in hopes to increase performance. We did not compact the FGDBs in the dev tier. Hoping my testing of the above GP history also should test out if the compacting of these two data sources (and possibly subsequent truncate and appends) have caused any issues.
• I have added an additional max instance to each service on the production server. Doesn't seem to have had much affect from today's initial testing.

Good suggestion on the antivirus as well- I'll have my IT folks confirm that nothing new has been setup or that nothing is regularly running or scanning the directories AGS needs to access for SOC operation. 

Any other ideas or thoughts are welcome. Thanks all!

-Rex

[DEWright_CA]

I would also consider looking at your Performance Monitor; get a look a the amount of memory each SOC is using and the amount of total SOCs running. Considering the performance hit of apps/service hitting the local pagefile; you may also be seeing a memory/swap contention type issue.

I usually shoot to have 128-256mb of RAM for each expected SOC; and when I am looking at setting up services for things like GeoProc I will have these create/write/delete a intermediary fGDB and avoid using InProc layers/features.