Metrics and Alerts on Shared Instances

[NickN]

I'm new to Monitor and recently deployed the new version to our production environment shortly after release. I'm curious what best practice is, or what others are doing for services that are on the shared instance pool.

It seems whenever one of these services is queried the "Instances Saturation Percent" metric goes to 100% and by default that triggers an alert.

Do most people disable alerting on these, or is there some other way I should be handling these?

Thanks!

[GeoJosh]

Hello,

I'm curious what ArcGIS Server's "Number of shared instances per machine" setting is currently set to. You can check this by signing into ArcGIS Server Manager > Site > Settings.

The default is 4 and when I query a service that is set to "Shared", the "Instances Saturation Percentage" metric in Monitor reflects between 15% and 30% which is what I would expect. If you are seeing that this metric is spiking to 100% every time a service is queried, this may be a sign that it should be increased.

Josh

[NickN]

Hi Josh,

Our shared instances is set to 8, and we just have a single machine site. Maybe I'm just coincidentally viewing it when all instances are handling active requests (which may include other services in the pool)

This is what the graph looks like for a service that's in the shared pool:

As you can see it's always bouncing from 0% to 100%

[MKR]

Hi Nick,

I am guessing that is a shared instance percent of one particular service?

An alternative to increasing the number of shared instances would be to try to switch the service to dedicated, especially if its a service that is used a lot and you don't want the initial start-up lag then set it to 1 and 2 and review again the next day (note: services set as dedicated and 1 minimum will always be running and consuming resources on the server).

In our environments I have created a graph showing all instances that are spiking to 100% use and review this regularly.

What version of enterprise are you on? 10.9.1?

[NickN]

Yeah I'm on 10.9.1.

This seems to be the case for any service on the shared pool. I ran into this as I was attempting to move some infrequently used services into the shared pool. The documentation suggested that good candidates were: "for most deployments, this means fewer than one service request per minute on average." So that's what I started with when I ran into this issue.

For example the two charts below are for a single service in the shared pool. You can see the smaller spikes around 7:50, 8:00, 8:30, 8:35 all were where 2 requests were received, yet the instance saturation at the same time was 100%.

Basically I never see a saturation percent at any value other than 0 or 100 on a service in the shared pool.

[MKR]

Are your services being published as dedicated and then you are switching to shared or is this a legacy from an upgrade that services were all set to dedicated?

Looks like the same issue we have on our environments with shared services. I would personally log a ticket with ESRI and ask them to investigate. I had no luck but I think its an issue with 10.9.1, as it happened immediately post upgrade to 10.9.1. Workaround for me, is to switch services having this issue to dedicated.

[NickN]

That's an interesting thought. I do believe all of these services were published as dedicated then switched to shared because I, among other people, never remember to set it to shared in the config before publishing 😅

I might test that out and publish a new service configured to use the shared pool and see if it reports differently.

[NickN · Accepted solution]

Yeah that definitely looks like the issue. I created a new service and configured it to the shared pool in the Share Web Layer pane in Pro and that graph looks like what I would expect:

I also noticed in the Overview screen in the Specifications section the new service it shows 8 in the Instance Limits section:

While all the older services that were switched from dedicated to shared have the Instance Limits blanked out:

And it's only those one's that are blanked out like that where the saturation graph bounces from 0 to 100%. So that got me thinking.

I switched the service back to Dedicated in ArcGIS Server Manager. Waited for the item in ArcGIS Monitor to indicate Dedicated again, then republished it and specified Shared in Pro. Then when Monitor refreshed it showed Shared with Min 8, Max 8 in the specs box as well as showing the correct saturation percent...so there's a work-around to fix it 🙂

Thanks for pointing me in the right direction, @MKR 

[NickN]

I can't help myself, so I continued to dig. Some of the services I switched to shared show 0 for both the Min and Max instances per machine when you look at their properties in the server admin directory. These are the ones that bounce between 0 and 100. So to test, I edited the min/max instance for one of my shared pool services to 1/2 and now it's reporting properly in Monitor.

For whatever reason those get set to 0 sometimes when moving to a shared instance and I think that causes issues in how Monitor reports the instance saturation. Changing these values in the admin dir is probably not a supported workflow but if it's part of the shared pool I don't think these numbers matter. I also have a ticket open with support to see what they say.

[SteveMcCarthy]

We have our new services defaulting to Shared when they are created in our 10.9.1 Environment.