We have an existing ArcGIS Server (AGS) 10.0 solution that is hosting close to 1,000 mapping services. We have been working on an upgrade to this environment to 10.2.1 for a few months now and we are having a hard time getting a stable environment. These services have light use, and our program requirements are to have an environment that can handle large amounts of services with little use. In the AGS 10.0 space we would set all services to 'low' isolation with 8 threads/instance. We also had 90% of our services set to 0 min instances/node to save on memory. Below is a summary of our approaches and where we are today. I'm posting this to the community for information, and I am really interested in some feedback and or recommendations to make this move forward for our organization.
Background on deployment:
We have a few arcgis server deployments that look just like this and are all running fairly stable and with decent performance.
Approach 1: Try to mirror (as close as possible) our 10.0 deployment methodology 1:1
our first problem we ran into was publishing services with 0 instances/node. Esri confirmed 2 'bugs':
#NIM100965 GLOCK files in arcgisserver\config-store\lock folder become frozen when stop/start a service from admin with 0 minimum instances and refreshing the wsdl site
#NIM100306 : In ArcGIS Server 10.2.1, service with 'Minimum Instances' parameter set to 0 gets published with errors on a non-Default cluster
So... that required us to publish all of our services with at least 1 min instance per node. At 1,000 services that means we needed 100-125GB of ram for all the ArcSOC.exe processes running without any future room for growth....
Approach 2: Double the RAM on the AGS Nodes
The file-server crash was clearly caused by publishing a large amounts of services to this new arcgis server environment. We caused our clustered file servers to crash 3 separate times all during this publishing workflow. We had no choice but to isolate this config-store/directories to an alternate location. We moved it to a small web-server to see if we could simulate the crashes there and continue moving forward. So far it has not crashed that server since.
During bootups, with the AGS node hosting all the services, the service startup time was consistently between 20 and 25 minutes. We were able to find a start-up timeout setting at each service that was set to 300 seconds (5 minutes) by default. we set that to 1800 seconds (30 minutes) to try and get these machines to start-up properly. What was happening is that all the arcsoc.exe processes would build and build until some point they would all start disappearing.
In the meantime, we also reviewed the ArcGIS 10.2.2 Issues Addressed List which indicated:
NIM099289 Performance degradation in ArcGIS Server when the location of the configuration store is set to a network shared location (UNC).
We asked our Esri contacts for more information regarding this bug fix and basically got this:
…our product lead did provide the following as to what updates we made to address the following areas of concern listed inNIM099289:
- 1. The Services Directory
- 2. Server Manger
- 3. Publishing/restarting services
- 4. Desktop
- 5. Diagnostics
ArcGIS Server was slow generating a list of services in multiple places in the software. Before this change, ArcGIS Server would read from disk all services in a folder every time the a list of services was needed - this happened in the services directory, the manager, ArcCatalog, etc. This is normally not that bad, but if you have many many services in a folder, and you have a high number of requests, and your UNC/network is not the fastest, then this can become very slow. Instead we remember the services in a folder and only update our memory when they have changed.
Approach 3: Upgrade to 10.2.2 and add 3 more servers
This is the closest we have gotten. At least all services are published. Unfortunately it is not very stable. We continually receive a lot of errors, here is a brief summary:
Level Message Source Code Process Thread SEVERE Instance of the service '<FOLDER>/<SERVICE>.MapServer' crashed. Please see if an error report was generated in 'C:\arcgisserver\logs\SERVERNAME.DOMAINNAME\errorreports'. To send an error report to Esri, compose an e-mail to ArcGISErrorReport@esri.com and attach the error report file. Server 8252 440 1 SEVERE The primary site administrator '<PSA NAME>' exceeded the maximum number of failed login attempts allowed by ArcGIS Server and has been locked out of the system. Admin 7123 3720 1 SEVERE ServiceCatalog failed to process request. AutomationException: 0xc00cee3a - Server 8259 3136 3373 SEVERE Error while processing catalog request. AutomationException: null Server 7802 3568 17 SEVERE Failed to return security configuration. Another administrative operation is currently accessing the store. Please try again later. Admin 6618 3812 56 SEVERE Failed to compute the privilege for the user 'f7h/12VDDd0QS2ZGGBFLFmTCK1pvuUP1ezvgfUMOPgY='. Another administrative operation is currently accessing the store. Please try again later. Admin 6617 3248 1 SEVERE Unable to instantiate class for xml schema type: CIMDEGeographicFeatureLayer <FOLDER>/<SERVICE>.MapServer 50000 49344 29764 SEVERE Invalid xml registry file: c:\program files\arcgis\server\bin\XmlSupport.dat <FOLDER>/<SERVICE>.MapServer 50001 49344 29764 SEVERE Unable to instantiate class for xml schema type: CIMGISProject <FOLDER>/<SERVICE>.MapServer 50000 49344 29764 SEVERE Invalid xml registry file: c:\program files\arcgis\server\bin\XmlSupport.dat <FOLDER>/<SERVICE>.MapServer 50001 49344 29764 SEVERE Unable to instantiate class for xml schema type: CIMDocumentInfo <FOLDER>/<SERVICE>.MapServer 50000 49344 29764 SEVERE Invalid xml registry file: c:\program files\arcgis\server\bin\XmlSupport.dat <FOLDER>/<SERVICE>.MapServer 50001 49344 29764 SEVERE Failed to initialize server object '<FOLDER>/<SERVICE>': 0x80043007: Server 8003 30832 17
Other observations:
Error: Error exporting map
Code: 500
Options for the future
We have an existing site with the ArcGIS SOM instance name of 'arcgis'. These 1,000 services are running in that 10.0 site for the past few years. Users have interacted with this using a URL like: http://www.example.com/arcgis/rest/services/<FOLDER>/<MapService>/MapServer
We are trying to host all these same services so that users accessing this URL will be un-impacted. If we cannot, we will switch to 1 server in 1 cluster in 1 site (and instead have 7 sites). We will then be re-publishing all our content to individual sites but will have different URL's:
http://www.example.com/arcgis1/rest/services/<FOLDER>/<MapService>/MapServer
http://www.example.com/arcgis2/rest/services/<FOLDER>/<MapService>/MapServer
...
...
http://www.example.com/arcgisN/rest/services/<FOLDER>/<MapService>/MapServer
We would have extensive amount of work to either (or both) communicate all the new URL's to our end users (and update all metadata, products, documentation, and content management systems to point to the new URL's) and/or build URL Re-direct (or URL Re-write) rules for all the legacy services. Neither of two options are ideal, but right now we seem to have exhausted all other options.
Hopefully this will help other users while they troubleshoot thier arcserver deployment. Any ideas are greatly appreciated with our strategy to make this better. Thanks!
ESRI was insistent that it needed to be set to system managed with enough RAM available and then some. When we switched from it being hard set at 16GB to system managed, rebooted we saw immediate improvement in performance and stability.
Sent via the Samsung Galaxy Note® 3, an AT&T 4G LTE smartphone
Interesting.
I'll relay this to IT and see if it's something that can be done easily.
I feel like we make changes every day to this thing, but we need to resolve this...
thanks again for all your help!
Aaron:
This is very interesting.
Too bad Esri dev wouldn't give out more info.
If the server has enough RAM it should never have to swap to Virtual Memory.
But it can be easy to use up RAM in ArcGIS.
Maybe that's what it's all about.
Even setting a big limit like 24GB, which would seem sufficient,
really isn't and with the system managing the Swap file, it can grow as big as needed.
I had one question, are you using virtual servers?
Paul,
Yes, we are using HyperV. OS is MS 2012. We also try to do a weekly reboot of all our servers as I have heard mumblings around the community of chronic hung soc processes and memory leaks with the 2 java platform SE binary processes that run.
Aaron
That's right, I recall prior discussions about Hyper V... funny they weren't showing up in my browser.
We also do a reboot due to those exact issues.
We also have cmd windows that are left hanging around by pskill processes.
Most of our servers are still on 10.1. & MSrvr 2008 R2
I do an automated reboot every two weeks which is usually sufficient.
Every once in a great while, our main server will still go south on us but there's no rhyme or reason to it.
It can happen ten days out, or 2 or 3 or ? days out....
How do you reboot your servers? Do you do it manually?
I set ours up via Task Manager but it was a real pain to get it working.
Actually ESRI (the guys from the Developer Summit 2016 say, for best practices, to have MIN Instance set to MAX if its a service that is heavily used. when min is a low digit (e.g. 1) and max is a larger one (e.g 5) when the demand is there, takes time for ArcSOC to build the instance... thus Min = max = fastest possible availability.
As some have mentioned above, situations like this may be caused by an exhausted non-interactive desktop heap size. The following technical article speaks to this scenario and should be consulted:
Please be aware of the warning messages contained within this article, as modifying the non-interactive desktop heap size can cause other issues and should be carefully tested.
Increasing the non-interactive desktop heap size was in fact the solution for one of our customers.
He couldn't start more than 333 processes on his single-machine-site. After increasing the non-interactive desktop heap size, he was able to start more without errors.
Thanks Thomas for the your link by increase the desktop heap size it indeed solved our issue similar to what OP had. Originally I can see the max number of background process stay around 320 include about 300 SOC.exe, and that cause serials of issues on the server and publishing. By increasing to 1024 from 768, I saw the backgroud processes now growed up to 360 and more SOC.exe were able to start.
Just curious, is there a rough math behind the desktop heap size and the number of processes could be supported. E.g. by increasing the heap size every 256, it allows additional X number of processes?
thanks