Problem
We’re experiencing serious issues with ArcGIS Server 10.1 stability. Services are crashing very frequently especially geoprocessing and map services which contains rasters. We’re also experiencing error messages like:
(The error messages above were extracted from the Log Viewer in ArcGIS Server Manager)
After a server restart, the services will be stable for a couple of hours but after that everything start to give problems again – either being very slow to serve maps or services crashing completely without returning any results.
ArcGIS Server Configuration
ArcGIS Server Installed Patches/Updates:
VM Specifications
Other Info
One must note that the server’s resources are no way near the limit. The CPU is on average always in the 50-60% range when serving a request while there is approx. 6GB of free RAM available for use.
Questions
Attempted Solutions
Prior to sending this email, we tried some changes to try and solve the problem.
Any help please?
Is your config-store a UNC path or local path ?
Have you instances in high or low isolation ?
Check the c:\users\<arcgis user>\AppData\Local\Temp folder(s) to see if there are any remaining temp files from creating caches (especially raster/base maps) or GP services. These don't always get cleared out as they should, especially in pre 10.2 versions.
for 10.0 (maybe 10.1?), the <arcgis user> was typically ArcGISSOC and a sample of a folder left behind would be something similar to
C:\Users\ArcGISSOC\AppData\Local\Temp\L2ACF{7A69647C-7874-4AAF-8286-76A0232B7766}\L09
.....of course the info after "Temp\" will vary. These file may contain images, .bundle and/or .bundlex files.
For other versions, the <arcgis user> may be the user that is used to run the ArcGIS Server service, and may be more than one user account (as is the case with our 10.2.2 and 10.3 machines). So check all the \AppData\Local\Temp files for older data. Be aware that is the have locks on them, they may actually be "live" data.
I find that cleaning the old temp files out once in a while helps get rid of many strange behavior. It may not help with your issue, but good to check anyway.
If you have many services created and used the defaults to create those services, under the processing on then you have 1 to 2 SOC accounts per service running on your server, each SOC is ~64 - 120 mb in size. By having these defaults the server is keeping the space available for when the service is needed, the problem is maybe some of those services are only needed occasionally. I had a similar thing happen, you can change the minimum and maximum number of instances per server by going into ArcCatalog>GIS Servers>(Connect to your server).
Right click on each service and choose service properties>Pooling and change the minimum and maximum number of instances (I set the minimum to 0 and the max to 1, because the server will start with these numbers on a clustered environment and add more beyond this as needed). Now the SOC's go away until the service is hit. This should free the memory to be used, the services may not respond as quickly as you like, but should be faster than what you are experiencing, but there are other configuration and tuning settings that you can do to assist with this. I have over 50 map services and my memory was always being used up, now I do not have those issues just doing this. I hope it helps.