Afternoon slowdown of ArcMap, arcsoc.exe

[JayHodny]

We have noticed for many months an afternoon slowdown (~3:00pm on) when working in ArcMap (Pro does not seem to be as affected).  Now that we are working from home, the issue seems to plague us somewhat throughout the day.  Further, when I rec and post in ArcCatalog, I no longer see the status window, and when I assume the process is finished, I have to close AC with the Task Manager, and then reopen.  I can understand home connectivity, routers, bandwidth, etc. causing some of this slowdown, but I would not expect this when connected directly at our office.  We have ArcGIS for Server 10.7.1, enterprise geodatabases 10.7.1, and ArcMap at 10.7.1.  Pro is at its latest version also. 

We are not a large user of the platform, but do have services published to Server for editing in web maps and apps.  I noticed in reviewing the Stats in Server that one service (having a couple dozen editable layers) was getting the majority of the requests when the user was doing some editing.  I have no frame of reference (is this a lot of requests or unusual?), but I don't think this editing was causing the overall slowdown.  In Windows Task Manager on the Server machine, we did notice there were many (130+ as of this morning) arcsoc.exe instances(?) running.  All were using no CPU resources, but had consumed 99% of the memory.  My colleagues and I are outside of our wheelhouse, so we are asking for input on what could be causing this routine slowdown.  Many thanks!

[George_Thompson]

I would look at adding more memory to the Server machine. Especially if it is at 99%. I would wonder if the memory getting to 99% corresponds to performance issues with the data coming from ArcGIS Server.

What is the memory used in the morning, with no performance issues?

[JayHodny]

Hello George, thank you for you the quick response.  We have 16 gig of memory on the ArcGIS for Server machine, all of it dedicated to Server.  When our IT group looked this morning, when the 130+ arcsoc.exe instances were active, 88% of the memory was being consumed.  I don't know the answer to your question regarding memory use when the platform is not experiencing a slow down.  Regards, Jay

[George_Thompson]

I would say to start with adding more memory to the VM, try 32GB and see if that has any impact. Having the memory that high can lead to performance issues when the services "request" more resources.

How many services and what types of services are on the ArcGIS Server machine?

I am not focused on how many ArcSOC's at this time. What is the min/max for the services?

Can you make the min 0 for some of the services that are not utilized often to release resources?

Here is a good site that may help with some of the technical questions: Server Software Performance - GIS Wiki | The GIS Encyclopedia 

[JayHodny]

Hi George,  thank you again for your help.  I will get out IT group involved as they need to be the ones adjusting/adding the memory.  I am guessing about 100 feature and/or map services, though many are for data query.  We are going to set min=0 on all the services that don't see much activity later today.  Looking forward to reviewing the wiki doc.  Thank you, Jay

[George_Thompson]

I am not guaranteeing that will fix it, but it is a good place to start. Just make sure that you look at what the min. instances should be for a specific service. If it is set to 0, it will still work but there may be an initial delay when a request is sent to the service.

[George_Thompson]

Another item to look at would be the RDBMS server to see what the health looks like. What is the RDBMS being used (type, version , etc.)?

Is it on a separate machine from ArcGIS Server (I am guessing it is based on comments above)?