topic Re: Metrics and Alerts on Shared Instances in ArcGIS Monitor Questions

Metrics and Alerts on Shared Instances

NickN — Fri, 17 Mar 2023 16:15:17 GMT

I'm new to Monitor and recently deployed the new version to our production environment shortly after release. I'm curious what best practice is, or what others are doing for services that are on the shared instance pool.

It seems whenever one of these services is queried the "Instances Saturation Percent" metric goes to 100% and by default that triggers an alert.

Do most people disable alerting on these, or is there some other way I should be handling these?

Thanks!

Re: Metrics and Alerts on Shared Instances

GeoJosh — Fri, 17 Mar 2023 17:04:07 GMT

Hello,

I'm curious what ArcGIS Server's "Number of shared instances per machine" setting is currently set to. You can check this by signing into ArcGIS Server Manager > Site > Settings.

The default is 4 and when I query a service that is set to "Shared", the "Instances Saturation Percentage" metric in Monitor reflects between 15% and 30% which is what I would expect. If you are seeing that this metric is spiking to 100% every time a service is queried, this may be a sign that it should be increased.

Josh

Re: Metrics and Alerts on Shared Instances

NickN — Fri, 17 Mar 2023 19:06:28 GMT

Hi Josh,

Our shared instances is set to 8, and we just have a single machine site. Maybe I'm just coincidentally viewing it when all instances are handling active requests (which may include other services in the pool)

This is what the graph looks like for a service that's in the shared pool:

As you can see it's always bouncing from 0% to 100%

Re: Metrics and Alerts on Shared Instances

MKR — Mon, 20 Mar 2023 10:07:29 GMT

Hi Nick,

I am guessing that is a shared instance percent of one particular service?

An alternative to increasing the number of shared instances would be to try to switch the service to dedicated, especially if its a service that is used a lot and you don't want the initial start-up lag then set it to 1 and 2 and review again the next day (note: services set as dedicated and 1 minimum will always be running and consuming resources on the server).

In our environments I have created a graph showing all instances that are spiking to 100% use and review this regularly.

What version of enterprise are you on? 10.9.1?

Re: Metrics and Alerts on Shared Instances

NickN — Mon, 20 Mar 2023 13:51:32 GMT

Yeah I'm on 10.9.1.

This seems to be the case for any service on the shared pool. I ran into this as I was attempting to move some infrequently used services into the shared pool. The documentation suggested that good candidates were: "for most deployments, this means fewer than one service request per minute on average." So that's what I started with when I ran into this issue.

For example the two charts below are for a single service in the shared pool. You can see the smaller spikes around 7:50, 8:00, 8:30, 8:35 all were where 2 requests were received, yet the instance saturation at the same time was 100%.

Basically I never see a saturation percent at any value other than 0 or 100 on a service in the shared pool.

Re: Metrics and Alerts on Shared Instances

MKR — Tue, 21 Mar 2023 14:00:32 GMT

Are your services being published as dedicated and then you are switching to shared or is this a legacy from an upgrade that services were all set to dedicated?

Looks like the same issue we have on our environments with shared services. I would personally log a ticket with ESRI and ask them to investigate. I had no luck but I think its an issue with 10.9.1, as it happened immediately post upgrade to 10.9.1. Workaround for me, is to switch services having this issue to dedicated.

Re: Metrics and Alerts on Shared Instances

NickN — Tue, 21 Mar 2023 19:18:30 GMT

That's an interesting thought. I do believe all of these services were published as dedicated then switched to shared because I, among other people, never remember to set it to shared in the config before publishing 😅

I might test that out and publish a new service configured to use the shared pool and see if it reports differently.

Re: Metrics and Alerts on Shared Instances

NickN — Tue, 21 Mar 2023 19:58:38 GMT

Yeah that definitely looks like the issue. I created a new service and configured it to the shared pool in the Share Web Layer pane in Pro and that graph looks like what I would expect:

I also noticed in the Overview screen in the Specifications section the new service it shows 8 in the Instance Limits section:

While all the older services that were switched from dedicated to shared have the Instance Limits blanked out:

And it's only those one's that are blanked out like that where the saturation graph bounces from 0 to 100%. So that got me thinking.

I switched the service back to Dedicated in ArcGIS Server Manager. Waited for the item in ArcGIS Monitor to indicate Dedicated again, then republished it and specified Shared in Pro. Then when Monitor refreshed it showed Shared with Min 8, Max 8 in the specs box as well as showing the correct saturation percent...so there's a work-around to fix it 🙂

Thanks for pointing me in the right direction, @MKR

Re: Metrics and Alerts on Shared Instances

NickN — Tue, 21 Mar 2023 21:50:40 GMT

I can't help myself, so I continued to dig. Some of the services I switched to shared show 0 for both the Min and Max instances per machine when you look at their properties in the server admin directory. These are the ones that bounce between 0 and 100. So to test, I edited the min/max instance for one of my shared pool services to 1/2 and now it's reporting properly in Monitor.

For whatever reason those get set to 0 sometimes when moving to a shared instance and I think that causes issues in how Monitor reports the instance saturation. Changing these values in the admin dir is probably not a supported workflow but if it's part of the shared pool I don't think these numbers matter. I also have a ticket open with support to see what they say.

Re: Metrics and Alerts on Shared Instances

SteveMcCarthy — Wed, 22 Mar 2023 13:38:36 GMT

We have our new services defaulting to Shared when they are created in our 10.9.1 Environment.

Re: Metrics and Alerts on Shared Instances

MKR — Tue, 28 Mar 2023 09:59:42 GMT

Hi Nick,

Let me know how you get on with the support ticket on this. Would be interesting to see if this setting is causing an issue with services in 10.9.1 or if its just related to how ArcGIS Monitor is reporting it because of those instance values.

Thanks,

Re: Metrics and Alerts on Shared Instances

NickN — Tue, 28 Mar 2023 13:16:57 GMT

I also have an issue where I still see services in the Monitoring panel that have been deleted that they think is related to an issue with the Discovery observer which they think is also causing the issue I described in this thread. They're still investigating.

Re: Metrics and Alerts on Shared Instances

AndrewOrton — Mon, 11 Dec 2023 20:09:22 GMT

This was logged as BUG-000157186, and has been fixed in ArcGIS Monitor 2023.2.