We have an interesting/frustrating case happening- - likely will result in a support call but wanted to see if more experienced minds have any thoughts.
We have 2 feature classes updated every minute via GeoEvent (Fire Dept Unit and Call location/status). These feature classes are then published in a standard map service and used in a webmap > ops dashboard. The layers are in the map multiple times, with different filters/symbology for use in the dashboard. The refresh interval for these layers is set at 1 minute - so, this is a lot of calls to the service. Multiply this by the # of users viewing the dashboard, and things get tricky (up to 20 users at a time - testing with 5 people).
We initially had the map service published on a general-use ArcServer machine, with the max instances bumped up to 15. Caused all kinds of problems with the ArcServer service and server in general - typical behavior being the whole ArcServer instance seemed to crash - nothing available in maps, arc server manager not available on that machine - but the Windows ArcServer service never stopped, RAM/CPU never spiked, and we were sure that the max # instances total per arcserver installation was not exceeded (200, I believe is the functional cap) - server is a VM, 4 CPU and 16 GB.
Put the service on a brand new ArcServer VM - (2 CPU, 8 GB) - ramped up the max # instances to 30. All is fine until we get 3+ people using the dashboard, then ArcServer goes haywire. Never seeing all 30 instances 'in use' and the # running varies between 2 and 25 or so. Only other service running is the Publishing service.
The logs show a few items which don't point in any single direction (to my eyes at least):
SEVERE: MapServerObjectFactory failed to create an instance of MapServer.
SEVERE: Geodatabase error: Out of server memory (however could still connect to the SDE database in question via ArcMap/Catalog)
WARNING: A connection with the server could not be established - WinINet Error while using HTTPS security, 12029)
- this may be a symptom after the service starts to misbehave
Again - no overall CPU/RAM spiking or sitting at terribly high levels on the server.
So my question is, how do we handle this? Even more instances of the service? Fewer? Other service properties to consider? More power for the VM? SQLServer connection issues? (database can't handle the traffic?)
using server v 10.4.1, SDE version 10.1 (I know, soon to be upgraded)
Thanks - sorry for the lengthy post