How to treat Severe ArcGIS Server Logs for System/DynamicMappingHost.MapServer?

[Yogesh_Chavan]

We are facing the below severe error logs on ArcGIS Server.

Trying to understand what could be triggering this or is there some ways to minimize/fix these errors?

Any help on this would be highly appreciated.

[JakeSkinner · Accepted solution]

Hi @Yogesh_Chavan, are you services configured with Shared or Dedicated instances? 

I've seen this error before when all services are configured with Shared.  Try changing the services to Dedicated.

https://enterprise.arcgis.com/en/server/latest/administer/windows/configure-service-instance-setting...

[Yogesh_Chavan]

Thank you @JakeSkinner for your insights.

We have 94 services on the server and very few are using the dedicated pool and all the others are on shared pooling.

Do we have to change pooling for all the services?

[JakeSkinner]

I would highly recommend the more frequently used services be switched to dedicated.  I typically only recommend a few services be set to shared instance type.

[DougBrowning]

We are getting these error dumps as posted above also and "Failed to obtain item from data store for given id: ()" also.  The admins set us up to do everything as shared to start and not use dedicated.  As we migrate to the new server more and more we started getting these errors and really poor performance on some bigger layers like polygons and image services.  Some will not draw at all and just site there.  Often we get totally white maps. 

We doubled the number of shared instances but still having issues.  Seems like moving anything that has issues to dedicated has helped.  But some still did crash out saying out of resources when set to 1 and 2 (I am thinking 0 min 2 max is best?).  We have over 300 services now (but 50 some are just views) and have it set to 16 instances (was set on a 8 core with 50 for each (which I think is the default).  Server CPU under 30% and RAM under 40% used.  Its a Azure VM. 

They come every second or so.  Often we can hit one and its fine but hit a second and it starts in the log.  We do not have too many users on yet so not sure what will happen then.  Polygons in a Ops Dashboard make it the most mad.

I am thinking we wack-a-mole any service that is slow and change to dedicated?  Or you seem to say just put the little ones that are not used much to shared then ones we know are big or used a lot to dedicated?  And set to?

Not finding too much on this so any help is greatly appreciated!  Been on it a week already.

thanks for any info  @JakeSkinner  I think we may have met in the past.

[JakeSkinner]

@DougBrowning I highly recommend heavily used services to be set to Dedicated.  For the number of instances, there is no default answer.  I would recommend starting at a min of 1 and a max of 4.  Let the service receive requests for a some time (i.e. half day/full day) and take a look in ArcGIS Server Manager > Logs > Statistics.  You can view the number of requests that particular service is receiving. 

Check the Average Response Time statistic for the service. 

I typically recommend keeping the response to 2 seconds or less.  If you see excessive times, check the Service Maximum Running  Instances for the service:

If the Max Running Instances are maxed out (i.e. 4 instances) this is indicating clients are waiting for a instance to handle their requests, which means you should increase the max instances for this service.

[DougBrowning]

Thanks a lot @JakeSkinner  but some issues still I have questions on.

I cannot seem to run any of these reports.  It lets me change the drop down but nothing updates.  Is this a rights issue?  I am not a full admin.

I started changing our services to dedicated, 1 and 4 as you said and its much better.  But we still are getting this error in the log a lot.  Even with only 1 user and 1 out of 4 instances we get it.

Failed to obtain item from data store for given id: ()

Happening on pretty much every service we try.  We double checked, republished, etc but it just keeps coming up.  Is it a managed data store config issue maybe?  We have a lose admin team right now and hard to tell who setup what and if its correct.  Portal is 11.3.

Posts like https://community.esri.com/t5/publishing-and-managing-services-questions/arcgis-server-log-severe-qu...  point to a resources thing but we are only using 1 of 4.  Is it resources or some other data store issue?  The SDE this all connects to is super fast and no issues there.

Thanks a lot we are going a bit nuts here!  

[Scott-Shavnore]

@DougBrowning which version of ArcGIS Pro are you publishing the layers from? You mentioned Portal/Federated Server is 11.3, so thinking you may be at ArcGIS Pro 3.3 but do you have any minor version patches installed? If publishing from Pro 3.3.5, you could be encountering this bug https://support.esri.com/en-us/bug/publishing-from-arcgis-pro-335-to-arcgis-enterprise-suc-bug-00017...

It was recently reported as fixed in Pro 3.3.7, I plan to begin testing this to see if it resolves the same issue in our environment.

[DougBrowning]

I am 3.3.5 yes.  We have 3.5.2 now on our test builds but I have not tested yet (something big going on here).  Will report back when we republish. 

[berniejconnors]

I agree with @JakeSkinner.  We configure our map services in a similar way.  We reserve the use of shared instances for services we are quite certain will be used infrequently.  About 20 map services (or 25%) use the shared pool.

Bernie.