topic Re: ArcSOC.exe process is running at high cpu usage in Publishing and Managing Services Questions

ArcSOC.exe process is running at high cpu usage

ShaunConway — Wed, 05 Jun 2013 18:49:42 GMT

ArcGIS Server 10 .Net
Windows Server 2008R2 Standard
RAM 4GB
Intel Xeon 2.67GHz (2 processors)

We have noticed that there is a persistent ArcSOC.exe process running between 20-30% CPU usage on our windows 2008R2 machine. We have stopped all services on the server and believe that this remaining process is either dedicated to logging or directory management (based on esri documentation.) If the process is stopped, it immediately regenerates.

What we are trying to figure out is why this process is running at such a high CPU usage. This is our development machine, and the identical production machine is not experiencing the same high CPU usage. The settings for ArcGIS Server are identical. In testing, I set the logging to verbose expecting to see an increase in the CPU usage associated with this process, but none occurred. I am not sure what "directory management" is and am therefor struggling to test whether or not this is the cause.

Any suggestions or help would be greatly appreciated.

Re: ArcSOC.exe process is running at high cpu usage

nicogis — Fri, 07 Jun 2013 07:48:35 GMT

Have you installed last service pack?

Re: ArcSOC.exe process is running at high cpu usage

ShaunConway — Fri, 07 Jun 2013 11:40:14 GMT

Domenico,

Thank you for your response. We are running SP5 across the entire enterprise.

Thanks,
Shaun

Re: ArcSOC.exe process is running at high cpu usage

ShaunConway — Fri, 07 Jun 2013 15:43:21 GMT

Here is a video of the ArcSOC.exe process. Note that all the other ArcSOC.exe processes are at 0%. This is our test server and these services are idle.

" rel="nofollow" target="_blank">http://www.youtube.com/watch?v=OwEUn7l13mQ&feature=youtu.be[/video]

Re: ArcSOC.exe process is running at high cpu usage

nicogis — Mon, 10 Jun 2013 14:02:42 GMT

Have you details in event viewer Windows or log arcgis server?

Re: ArcSOC.exe process is running at high cpu usage

ShaunConway — Tue, 18 Jun 2013 13:35:40 GMT

There isn't any critical information in the windows event viewer or AGS logs.

Re: ArcSOC.exe process is running at high cpu usage

LakshmananVenkatesan — Tue, 17 Dec 2013 00:39:28 GMT

HI Shaun,

We are also facing similar kind of issue, did you got any information or updates on this problem

Re: ArcSOC.exe process is running at high cpu usage

RexRobichaux3 — Wed, 21 Nov 2018 22:35:42 GMT

Not to bring this back from the dead but- howdy Shaun!! Long time no see . This is a shot in the dark but why not:

1) Did you guys ever figure out what was causing the CPU spike on the AGS for some SOC processes?

2) If so, do you recall what the issue or culprit was?

We have just recently started seeing much higher than "normal" CPU utilization from 2-3 SOC processes (intermittent but reproducible with light service use) that just started in the last week. It's bringing our prod server to a near stand still and 100% utilization when there are several users hitting our apps from the services. Also nothing in the logs that jump out. Any help or ideas would be greatly appreciated- hope all is well in Lynchburg!

-Rex

Re: ArcSOC.exe process is running at high cpu usage

DEWright_CA — Wed, 21 Nov 2018 23:04:20 GMT

What kind of services are you running that you see this issue with? Composite Geocoders can cause a larger load due to the nested data access.

Also various Geoprocessing tools can cause a spike after various OS patches to address security exploits from earlier this year.

Also consider taking a look at your Anti-Virus software; some configurations can see each time a service or its data is accessed that a AV scan of the data/files will be triggered which can cause the service to spin in a waiting-to-load state.

Re: ArcSOC.exe process is running at high cpu usage

RexRobichaux3 — Mon, 26 Nov 2018 15:07:04 GMT

Thanks for the reply David. We are seeing these delays / CPU utilization issues from standard map services that are referenced in a series of 10.3.1 Portal web applications. What is strange is that this was never an issue before a few weeks ago (checked and no Windows updates have been applied / nor Esri), and I cannot reproduce this behavior on our dev tier which mimics our prod tier. In the dev tier (and previously in prod) I can get CPU to spike to ~30% with some panning around (especially high multi-family areas where parcel drawing is more resource intensive) but never much more than that. The same app, same zoom extend and area panning now in prod will spike to 80-90% CPU and stay there for a little while after the draw is complete.

The only things that are different between prod and dev is:

Our prod server (VM ware, 16GB RAM, 4 core hyper VM with Windows 2012R2, Intel Xeon E5-2695 V4) is a new "clone" of our old VM server as of a month ago. However IT has assured us that from a hardware perspective, our specs are identical. The dev machine has identical specs as well.
Our spatial data that is used for web app consumption come from two file geodatabases that also live on the respective servers. We recently switched from using one-way child to parent replication to update these file geodatabases (from our production enterprise geodatabase) bi-weekly, to a python script that truncates and appends all of the data from our Prod Oracle EGDB.
- Im currently in the process of making new copies of these FGDBs and deleting the GP history / metadata that has accumulated due to this recurring truncate and append. Hoping to test performance from the new FGDBs tonight.
We also compacted these two FGDBs a few weeks ago in hopes to increase performance. We did not compact the FGDBs in the dev tier. Hoping my testing of the above GP history also should test out if the compacting of these two data sources (and possibly subsequent truncate and appends) have caused any issues.
I have added an additional max instance to each service on the production server. Doesn't seem to have had much affect from today's initial testing.

Good suggestion on the antivirus as well- I'll have my IT folks confirm that nothing new has been setup or that nothing is regularly running or scanning the directories AGS needs to access for SOC operation.

Any other ideas or thoughts are welcome. Thanks all!

-Rex

Re: ArcSOC.exe process is running at high cpu usage

ShaunConway — Tue, 27 Nov 2018 13:52:17 GMT

Hey Rex!

Hope all is going well. It's been a while, so I don't remember exactly what we did to resolve this, if anything. To be honest, we may have just stuck it out until an AGS upgrade, but our issue was of course on development which allowed us that flexibility. There were are few issues we had going on around this time, so here are some things we did around that time to resolve issues ... maybe one of them will be useful?

- Created a new, identical map service and used that in place of the one with the issue

- Restarted AGS and then the server itself (I know this is an obvious one, but it's amazing how many times it has resolved our issues.)

- Republished the service, removing one layer at a time, to determine if one layer was the culprit or if it was an issue with the service/AGS.

- Republished the service pointing to a copy of the data. Copy of the data wasn't a copy/paste or import, but a clean feature class, import schema, then append.

Also I'll double down on David's suggestion of Anti-Virus software. I know that was a major issue for us at one time in a previous AGS release. Simple rule change cleared it all out.

Sorry I couldn't be of more help!

Shaun

Re: ArcSOC.exe process is running at high cpu usage

DEWright_CA — Tue, 27 Nov 2018 17:36:28 GMT

I would also consider looking at your Performance Monitor; get a look a the amount of memory each SOC is using and the amount of total SOCs running. Considering the performance hit of apps/service hitting the local pagefile; you may also be seeing a memory/swap contention type issue.

I usually shoot to have 128-256mb of RAM for each expected SOC; and when I am looking at setting up services for things like GeoProc I will have these create/write/delete a intermediary fGDB and avoid using InProc layers/features.

Re: ArcSOC.exe process is running at high cpu usage

RexRobichaux3 — Fri, 30 Nov 2018 21:02:58 GMT

Thanks again Shaun and David for the replies and help! I have a few updates to pass along just in case anyone suffers this CPU behavior going forward. It looks like there are a myriad of issues causing our problems for our production server. The steps we have taken and the issues that have been identified:

We migrated our VM server to a new host with better processing resources. Our production server seemed to like this as even though we still had higher-than-we-used-to-have levels of CPU utilization, it was no longer maxing out resources and bringing apps and services to a haulty- so we likely had some underlying hardware issues.
We further improved performance by adding all of the arcgis server directories to the exemption list of (both) antivirus applications running on the server. These same antiv applications run on all of our servers including dev, but they seemed to be going a little wild on production- IT is looking into why. With all of this done CicsoAMP still seems to be going crazy so we are looking into why CiscoAMP is having issues in this environment. We have had to completely deactivate CiscoAMP for the time being until we figure out why. The directories we admittedly over-zealously added:
- c:\arcgisserver
- c:\Program Files\ArcGIS\Server
- c:\Python27
- c:\Program Files\Esri
- c:\ Program Files\Common Files\ArcGIS
- c:\app\Oracle (our oracle client on the server)
- c:\Webservices\Data (our file geodatabase storage directory for GIS published data)
- e:\arcgiscache (cached aerials directory on the server)
It appears that using the search widget in any app, without specifying a map service layer, is a major contributor to the CPU usage. As any one of our apps has between 15 and 25 searchable layers, this is not really surprising. Therefore we will have to continue with more end-user education that when using the search widget, the layer of interest needs to be specified. We have always been pretty religious with this messaging but there maybe a new hire or app user who doesn't know this or has forgotten that is unknowingly really slowing things down. I also have a case open with Esri support to verify that this issue is expected behavior and not a bug.

Hope this helps guys- if we figure out anything else I'll update this. Thanks again for the info and assistance!

-Rex