Unstable ArcGIS Server 10.1

[JohannBorg1]

Problem

We’re experiencing serious issues with ArcGIS Server 10.1 stability. Services are crashing very frequently especially geoprocessing and map services which contains rasters. We’re also experiencing error messages like:

• “Unknown Container Exception: org.apache.openejb.client.RemoteFailoverException: Cannot complete request.  Retry attempted on 1 servers; nested exception is: \n\tjava.io.IOException: Cannot deternmine server protocol version: Received null/0.0”
• Internal Server Error. Error handling service request : a20c5357-665b-4202-8c45-77d932772de6 [Source: REST]
• Failed to construct instance of service 'xxxxxxxxxx'. a20c5357-665b-4202-8c45-77d932772de6 [Source: Server]
• Instance of the service 'xxxxxxxxxx' crashed. [Source: Server]
• Internal Server Error. Error handling service request : Cannot obtain a free instance.; nested exception is: com.esri.arcgis.discovery.ejb.ArcGISServerEJBException: com.esri.arcgis.discovery.servicelib.AGSException: bcabe55e-552a-4085-90f4-faaa9ac492a4 [Source: REST]

(The error messages above were extracted from the Log Viewer in ArcGIS Server Manager)

After a server restart, the services will be stable for a couple of hours but after that everything start to give problems again – either being very slow to serve maps or services crashing completely without returning any results.

ArcGIS Server Configuration

• Current Version: 10.11
• Total Map Services Published: 15 (8 of which contains raster images)
• Total Feature Services Published: 7 (1 of which contains raster images)
• Total Geoprocessing Services Published: 21
• ArcGIS Server is accessible from client application through the Web Adaptor (running on IIS)
• All data is stored in two separate geodatabases, which geodatabases are residing on another dedicated data server
• Four data stores are registered with ArcGIS Server – 2 Databases and 2 Folders
• No Extensions are registered with ArcGIS Server
• Web Adaptor and ArcGIS Server are installed on the same Virtual Machine
• Licence Level: ArcGIS for Server Standard

ArcGIS Server Installed Patches/Updates:

• Service Pack 1
• QIP-S-SEC-JAN2015-Patch
• S-LS-Patch
• S-QIP-Patch
• WAI-SEC-Patch

VM Specifications

• OS: Windows Server 2008 R2 Enterprise
• Storage Space: 500GB (currently 428 GB free)
• RAM: 16GB
• CPU: Intel Xeon @ 2.6GHz (quad-core)

Other Info

One must note that the server’s resources are no way near the limit. The CPU is on average always in the 50-60% range when serving a request while there is approx. 6GB of free RAM available for use.

Questions

1. From my investigations it looks like it can be a memory problem i.e. ArcGIS Services not utilizing enough memory. Is there any limit of how much memory is used by each service?
2. Could it be that the amount of geoprocessing services (21) published on the server are somehow helping to create this issue?
3. ArcSDE was not installed since from ArcGIS v10, ArcGIS Desktop supports Direct Connection. Does this make any difference?
4. One of the geoprocessing service is a custom ArcPy script which is using the ExportToPDF tool. This was working perfectly until some time ago. Lately it started giving problems as well namely:
   1. If a webmap contains a raster image, the PDF is generated but no rasters are included in the map – only vectors
   2. If a webmap contains a couple of vector datasets, the PDF generation will fail

Attempted Solutions

Prior to sending this email, we tried some changes to try and solve the problem.

1. Create a new geodatabase and point a new service to it
2. Changed the App server max heap size and the SOC max heap size as described in this forum (https://community.esri.com/thread/111267#416933) [Now they are back to the original values]
3. Re-created the config-store
4. We’re trying to avoid re-building the site (just for the sake of trying) because this means downtime for our customer BUT if this is a solution we can schedule it.

Any help please?

[nicogis]

Is your config-store a UNC path or local path ?

Have you instances in high or low isolation ?

[RebeccaStrauch__GISP]

Check the c:\users\<arcgis user>\AppData\Local\Temp folder(s) to see if there are any remaining temp files from creating caches (especially raster/base maps) or GP services.  These don't always get cleared out as they should, especially in pre 10.2 versions.

for 10.0 (maybe 10.1?), the <arcgis user> was typically ArcGISSOC and a sample of a folder left behind would be something similar to

   C:\Users\ArcGISSOC\AppData\Local\Temp\L2ACF{7A69647C-7874-4AAF-8286-76A0232B7766}\L09

.....of course the info after "Temp\" will vary. These file may contain images, .bundle and/or .bundlex files.

For other versions, the <arcgis user> may be the user that is used to run the ArcGIS Server service, and may be more than one user account (as is the case with our 10.2.2 and 10.3 machines).  So check all the   \AppData\Local\Temp files for older data.  Be aware that is the have locks on them, they may actually be "live" data.

I find that cleaning the old temp files out once in a while helps get rid of many strange behavior.   It may not help with your issue, but good to check anyway.

[ChadKopplin]

If you have many services created and used the defaults to create those services, under the processing on then you have 1 to 2 SOC accounts per service running on your server, each SOC is ~64 - 120 mb in size.  By having these defaults the server is keeping the space available for when the service is needed, the problem is maybe some of those services are only needed occasionally.  I had a similar thing happen, you can change the minimum and maximum number of instances per server by going into ArcCatalog>GIS Servers>(Connect to your server).

Right click on each service and choose service properties>Pooling and change the minimum and maximum number of instances (I set the minimum to 0 and the max to 1, because the server will start with these numbers on a clustered environment and add more beyond this as needed).  Now the SOC's go away until the service is hit.  This should free the memory to be used, the services may not respond as quickly as you like, but should be faster than what you are experiencing, but there are other configuration and tuning settings that you can do to assist with this.  I have over 50 map services and my memory was always being used up, now I do not have those issues just doing this.  I hope it helps.