Layer causing high ArcSOC EXE load

[abureaux]

Hello,

I have a strange one. Maybe someone has dealt with this in the past.

One of my larger layers (a few hundred thousand points) is causing issues all of a sudden.

• This layer previously had no issues.
• The problems began a couple days ago out of the blue (no layer chances, server updates, new processes, etc.).
• Now, whenever the web app which contains the layer is opened, the layer immediately tops out ArcSOC EXE, and the layer in question doesn't work.
• The layer is set to a dedicated instance pool.
• Currently, it has min 3 max 3 instances. I have played with various configurations as well (0/1, 1/1, 0/2, 1/3, etc.)
• There is a scale view set, so the layer isn't rendering when the web app opens. There are also two copies of the web app atm, built in WAB and EB. Doesn't matter which version is used.
• The layer was republished form ArcPro last night.
• Server has 4 physical and 8 Virtual cores. Resources in general aren't an issue right now for this machine.

Before opening web app:

After opening web app (the layer also doesn't work properly in the web app - it just does this while the web app is open):

Again, everything was fine a few days ago. This is very abnormal behaviour. Typically, using the web apps wouldn't even register on the server.

[JohnSteed1]

Hello abureaux

Have you tried changing the max_connections in /opt/arcgis/datastore/usr/arcgisdatastore/pgdata/postgresql.conf to a higher number.  I think the automatic setting is around 300.

You can then use the ‘changedbproperties.sh’ to see the change reflected in the postgresql.conf.

However, based on what you are showing here, it might be a "javaHeapSize" issue, which can be modified using the Edit Service operation in the ArcGIS Server Administrator Director. 

Good luck.

[ChristopherCounsell]

Esri advises against configuring the postgres data store or internal files for the ArcGIS Data Store. If something went wrong technical support might not help you.  

Abureaux also appears to be looking at referenced services not the ArcGIS Data Store, as you don't have control over the service pooling for hosted feature services. 

[ChristopherCounsell]

This is a tricky one to troubleshoot without exploring your environment. 

You can likely rule out some general optimization issues if the issue was not occurring previously, and a few hundred thousand points is not a lot. I'd recommend testing the service in a simpler environment to help remove components i.e. query REST API over opening in Map Viewer.]

Some things you could take a look at:

• ArcGIS Server logs (anything popping up for this service?)
• General service optimization, what you are doing with the service. Look at things like search, field indexes, projection, max return count, the heap size, if caching is being used
• AntiVirus. Some users highlight it as a high driver of CPU (where memory can be a hog for general ArcGIS usage)
• Environment - do you have enough disk space? Recently patched / updates current? Database healthy?

https://community.esri.com/t5/publishing-and-managing-services-questions/arcsoc-exe-process-is-runni...

[Dan_Brumm]

Had a similar experience, it ended up being the spatial index. Once I ran a reindex everything went back to normal. 

[abureaux · Accepted solution]

Over this last weekend, I rebuilt the webmap. I have since been monitoring server load, and things seem to be working again... In the end, I didn't change any settings. This was really strange.

This same layer was badly corrupt earlier in the year due to an ODBC connection issue. It took several days to repair the connection and re-publish the layer. After that republish, I noticed the layer appeared a little different than it did originally. But since everything was working again, and we had quite the backlog, we decided to leave well-enough alone. Maybe one of our routine layer operations just finally pushed it over the edge?

[k1etus]

I used to use 2 max 0 min till someone complained. Now I use shared instances till someone complains.

Shared instances have significantly lightened the load on the server plus I’m rarely managing arcsocs now.

There are a lot of variables to consider here. I just switched everything to shared and studied the logs. Services with frequent requests were switched back to dedicated with 2-0 and adjusted as needed.

[k1etus]

---

@k1etus wrote:

I used to use 2 max 0 min till someone complained. Now I use shared instances till someone complains.

Shared instances have significantly lightened the load on the server plus I’m rarely managing arcsocs now.

There are a lot of variables to consider here. I just switched everything to shared and studied the logs. Services with frequent requests were switched back to dedicated with 2-0 and adjusted as needed. https://omegle.onl/  

---

I got this,...