Service behaves differently with shared instance pool vs. dedicated instances

[AllenScully2]

Just resolved a major outage at my organization - the error involved changing the map service in question for using dedicated instances to using the shared instance pool.  

(Published to ArcGIS Server 10.9.1 using Pro 3.0)

Details below, but my understanding is that dedicated vs shared instance pool is purely about resource usage on the server, and should never impact the capabilities or behavior of a map service outside of the resources used. 

Brief summary - we have a custom ArcMap add-in to use for some complex editing to enforce data quality and simplify editing for non-ArcMap experts.   The tool consumes a Map Service and Geodata service (named the same thing) to allow users to check out data for editing, then check it back in.  The tool was broken after a migration to newer OS and Arc Enterprise versions and infrastructure (Win 2019 and Enterprise 10.9.1, from 2012 and 10.7.1 respectively).  We figured there were just some inherent compatibility issues as the tool is older, custom, and the services are published through Pro.

The error the tool returns when we use dedicated instances on the map service is: 

"Error getting unique IDs from map server: value does not fall in expected range"

However, the tool will allow users to check out data and edit/check in when we set the map service to use the Shared Instance Pool vs. the dedicated pool.  

the related Geodata service is using dedicated instances and that does not seem to be causing any issues.

This feels very much like a bug, or maybe just a sign of creeping compatibility issues in our infrastructure.  

But I am looking to confirm that Shared vs. Dedicated only comes in to play for service performance tuning and resource usage, and should not impact functionality.

I would have never thought that would be the cause of our problem.

Thanks - 

Allen

[MarceloMarques]

Adding these links to help with this discussion.

Tune and configure services—ArcGIS Server | Documentation for ArcGIS Enterprise

Configure service instance settings—ArcGIS Server | Documentation for ArcGIS Enterprise

[AllenScully2]

Thanks Marcelo - 

That's interesting - that's more or less matches what my understanding was. 

So our tool ONLY works using the Shared Instance pool, which seems counter-intuitive.  It is just a map service with no extra functionality - and we had it set for dedicated with a higher number of max instances due to an anticipated high usage period coming up.

This seems like something that is not happening not by design, but I worry that our set up is too customized to really report this as a bug.  

[MarceloMarques]

Indeed, also keep an eye on the server cpu and memory utilization, the dedicated service might have started to fail because the machine was running out of memory, the server needs enough free memory to accommodate the min-max number of SOC processes for the dedicate service, if the system is under heavy load and other services are consuming more memory then dedicated services might not be able to spin more SOC processes based on the min-max number that is configured. Hence, once you switched the service from dedicated instances to shared instances the system start to respond again. This is definitely something that you can keep an eye on to see how server memory is been used. Esri ArcGIS Monitor is a tool that can be used to monitor the server cpu, memory, disk but also to monitor the ArcGIS Server Services and you can create email alerts if a particular Service crosses a threshold metric value that you define, excellent tool to monitor ArcGIS Enterprise deployments. 
Introduction to ArcGIS Monitor—ArcGIS Monitor | Documentation

[Laura]

Is there a good way to know how many instances to set it as per machine?

[A_Wyn_Jones]

I believe the error message you're getting is due to a publishing incompatibility.

You should use ArcGIS Pro 2.9.x to publish to Enterprise 10.9.1, please see: https://pro.arcgis.com/en/pro-app/latest/help/analysis/geoprocessing/share-analysis/geoprocessing-se...

This may affect the performance of your geoprocessing tool as the checks for additional parameters are failing. 

You are correct that the user should experience the same performance with dedicated vs shared instances. Important to note if you're dedicated minimum instances are set to "0", the instance start-up (on user request) does take some time depending on the complexity of the tool/map service. 

[AllenScully2]

Thanks both for your responses - 

A Wyn - that is interesting regarding 2.9 + 10.9.1 for GP services, I did not know that.  In our case however, the service in question is a pretty vanilla map service - no feature access, not a GP service, no custom settings or anything unusual about it.  

As far as Marcelo's point about resource usage monitoring - we are aware of Monitor, I have used it before (though not in my current role) - but our server has not been under any suspiciously high load, we don't have a ton of services running, and it's a very well-provisioned VM (4 CPU, 32 GB - plenty for what this server does currently). 

In any case our immediate issue is resolved, I'm just confused by the solution (making a plain map service use Shared pool instead of dedicated) as it doesn't seem to be relevant at all.  But so it goes - our use of a custom-built add-in definitely complicates matters.   

Soon we'll have all users in Pro with a new version of the Add-in for Pro, and running 11.1 Enterprise, so this issue will likely go away once all of that is implemented.