topic Re: Generate an alert when accessing a stopped service in ArcGIS Monitor Questions

Generate an alert when accessing a stopped service

Lerman — Tue, 25 Jul 2023 09:51:53 GMT

Imagine a common scenario: a map service has stopped, perhaps due to some accident. However, the application is still using the service, and when it accesses the service, it gets a request response of 500:[ Error: Service XXX/MapServer not started Code: 500.] ArcGIS Server generates a log with the warning level:[ Unable to process request. Service XXX/MapServer not started.] In this case, I think ArcGIS Monitor should generate an alert so that the administrator can check the status of the service in time to avoid serious consequences. Surprisingly, ArcGIS Monitor does not generate any alerts for this case. （Is this correct? Is there an additional setting that I didn't notice?)

I tried to analyze this problem, and I think there are two reasons, the first reason is that ArcGIS Monitor doesn't count this kind of failed requests as failed requests, so even though I added the Request Failed(req) Metrics, it still didn't generate any alert. The second reason is that ArcGIS Monitor does not generate alerts for ArcGIS Server warning level logs. I know that earlier versions such as ArcGIS Monitor 10.8.1 could generate alerts for different levels of logs, however, for version 2023.1, I did not find such a setting.

Any suggestions please? I would like ArcGIS Monitor to be able to generate alerts for this situation.

Thank you!

Re: Generate an alert when accessing a stopped service

GeoJosh — Tue, 25 Jul 2023 13:19:08 GMT

Hello, ArcGIS Monitor does not yet open alerts based on component logs. This is a feature that has been requested by a few users and is on our road map. Please vote for this feature so the development team can gauge interest in it.

ArcGIS Monitor 2023.3 will introduce a new "disconnected" or "unreachable" status. This will be tracked in a new metric in which alert rules can be configured on.

Josh

Re: Generate an alert when accessing a stopped service

Lerman — Tue, 25 Jul 2023 13:57:44 GMT

@GeoJosh

Thank you very much, Josh! I really want to continue to confirm a few questions with you. ArcGIS Monitor does not count these requests as failed requests, so the current version of arcgis monitor cannot generate alerts, right? There is no other solution in the current version, right? In addition, when will version 2023.3 be released? Do you have any news? Thanks again!

Re: Generate an alert when accessing a stopped service

GeoJosh — Wed, 26 Jul 2023 15:10:35 GMT

Hi @Lerman,

ArcGIS Monitor does not count these requests as failed requests, so the current version of arcgis monitor cannot generate alerts, right? There is no other solution in the current version, right?

You can be notified via e-mail and webhook when a service is stopped by creating a new custom notification with the following configuration:

For "Operations", choose "Updated"
For ID, specify the ID of the service component.

In addition, when will version 2023.3 be released?

We have not yet settled on an exact release date for 2023.3.

Josh

Re: Generate an alert when accessing a stopped service

Lerman — Thu, 27 Jul 2023 03:01:12 GMT

Hello @GeoJosh ,

Thank you Josh! The solution you provided requires the use of a webhook right? I see in the help documentation that webhook supports geoprocessing and feature services, other service types are not supported. Thank you!

Re: Generate an alert when accessing a stopped service

GeoJosh — Thu, 27 Jul 2023 12:43:54 GMT

@Lerman, Can you point me to what documentation you are referring to?

Re: Generate an alert when accessing a stopped service

PhilLarkin1 — Thu, 27 Jul 2023 17:51:35 GMT

Consider voting for this idea:
https://community.esri.com/t5/arcgis-monitor-ideas/alert-on-the-status-of-a-service-stopping-starting/idi-p/1265854/jump-to/first-unread-message

Re: Generate an alert when accessing a stopped service

Lerman — Fri, 28 Jul 2023 01:15:11 GMT

Hi @GeoJosh , It's this document: https://enterprise.arcgis.com/en/portal/latest/administer/windows/about-arcgis-webhooks.htm

Re: Generate an alert when accessing a stopped service

Lerman — Fri, 28 Jul 2023 01:16:43 GMT

Okay, I voted. Thank you @PhilLarkin1

Re: Generate an alert when accessing a stopped service

GeoJosh — Fri, 28 Jul 2023 13:29:11 GMT

@Lerman, Thank you for clarifying.

Enterprise webhooks are different from ArcGIS Monitor webhooks. Webhooks in ArcGIS Monitor are one of two ways a notification can be delivered (the other being via e-mail). Notifications can be set on a variety of different resources in Monitor, including registered components and metrics. This means webhook notifications in ArcGIS Monitor are not limited to specific service types. A webhook payload will be emitted whenever the notification rules you define are triggered.

Please see this documentation for more on notifications and this documentation for more on the webhook payloads that ArcGIS Monitor emits.

Josh

Re: Generate an alert when accessing a stopped service

Lerman — Mon, 31 Jul 2023 02:09:02 GMT

Thank you @GeoJosh In arcgis monitor, it's all about asking me to fill in the url of the webhook, and I can't find any information about creating a webhook in the help of the arcgis monitor.

Re: Generate an alert when accessing a stopped service

GeoJosh — Mon, 31 Jul 2023 13:09:26 GMT

Hi @Lerman,

The webhook URL would come from the platform that you choose to implement with (e.g. Microsoft Teams, Slack, Integromat). Since there are many different platforms on which webhooks can be processed, this is not covered in the ArcGIS Monitor documentation.

Josh

Re: Generate an alert when accessing a stopped service

Lerman — Fri, 04 Aug 2023 07:17:53 GMT

Hello @GeoJosh ,

I'm having trouble finding articles on creating a webhook and using it together with a monitor. Do you have any resources on your side? The monitor supports webhooks, but there are few helpful instructions with detailed steps, which really increases the difficulty of use. Through Esri Support, I found this article (https://support.esri.com/en-us/knowledge-base/how-to-create-a-webhook-with-make-which-sends-an-email-000023798). Is this related to ArcGIS Monitor WebHook? I don't think it is. Thank you Josh!

Re: Generate an alert when accessing a stopped service

IrisHadarStltd — Wed, 09 Aug 2023 10:51:01 GMT

Hi ,
I tried to create the NOTIFICATION as you explained, but after saving, the system omits the condition of System state is stopped

Re: Generate an alert when accessing a stopped service

PatrickKielty — Tue, 26 Sep 2023 14:18:51 GMT

Did you ever find a solution to this issue? I am trying to do the exact same thing and the 'stopped' rules is gone after I click save. Also, not sure why we have to type the value into the value box when it should be populated. Thanks

Re: Generate an alert when accessing a stopped service

ChrisRobinsonCOF — Wed, 14 Feb 2024 19:31:55 GMT

Josh,

I was able to set up this notification like you have configured for a single service. Is it possible to set this up for multiple services instead of each individual service having to have a notification tied it? I was thinking maybe through a collection of all our maps services but when I choose the collection resource type, System State isn't presented as an option.

**UPDATE- Monitor version is 2023.3.1

Thank you,

Chris

Re: Generate an alert when accessing a stopped service

PhilLarkin1 — Wed, 14 Feb 2024 19:02:59 GMT

Are you on Monitor 2023.3?

You could try to set a condition based on Name + Contains + selection of service names. Or ID + Is Any Of + IDs you need.

Re: Generate an alert when accessing a stopped service

ChrisRobinsonCOF — Tue, 20 Feb 2024 20:48:30 GMT

Thank you Phil for the idea. Yes, I am on monitor 2023.3.1. That does seem to work for me but I have a few servers registered that have the same folder structure and map service name. If I stop a service on one server that has a duplicate name and folder structure from another, a notification doesn't get generated. If they are duplicated, the drop down list doesn't seem to recognize those duplicates by server so I am looking for a way to create a notification by server. I came up with this definition below, the logic seems sound but I am not getting a notification result if I stop a service:

Thank you,

Chris

Re: Generate an alert when accessing a stopped service

PhilLarkin1 — Tue, 20 Feb 2024 21:05:44 GMT

I've not used Internal Address. Curious if it is actually want you want to target. The description of that component is 'The common network address.'

https://doc.arcgis.com/en/monitor/latest/get-started/windows/arcgis-monitor-information-model.htm

Re: Generate an alert when accessing a stopped service

ScottCorwin — Tue, 07 May 2024 17:24:43 GMT

Did this feature make it into 2023.3.1? I'm trying to also create a notification, and get stuck with the system state not showing anything in the dropdown. I'm able to create the same expression in an analysis and see the status, but not in an event.