Generate an alert when accessing a stopped service

[Lerman]

Imagine a common scenario: a map service has stopped, perhaps due to some accident. However, the application is still using the service, and when it accesses the service, it gets a request response of 500:[ Error: Service XXX/MapServer not started Code: 500.]  ArcGIS Server generates a log with the warning level:[ Unable to process request. Service XXX/MapServer not started.]  In this case, I think ArcGIS Monitor should generate an alert so that the administrator can check the status of the service in time to avoid serious consequences. Surprisingly, ArcGIS Monitor does not generate any alerts for this case. （Is this correct? Is there an additional setting that I didn't notice?)

I tried to analyze this problem, and I think there are two reasons, the first reason is that ArcGIS Monitor doesn't count this kind of failed requests as failed requests, so even though I added the Request Failed(req) Metrics, it still didn't generate any alert. The second reason is that ArcGIS Monitor does not generate alerts for ArcGIS Server warning level logs. I know that earlier versions such as ArcGIS Monitor 10.8.1 could generate alerts for different levels of logs, however, for version 2023.1, I did not find such a setting.

Any suggestions please? I would like ArcGIS Monitor to be able to generate alerts for this situation.

Thank you!

[Lerman]

Okay, I voted. Thank you @PhilLarkin1 

[ScottCorwin]

Did this feature make it into 2023.3.1?  I'm trying to also create a notification, and get stuck with the system state not showing anything in the dropdown.  I'm able to create the same expression in an analysis and see the status, but not in an event. 

[GarrettMcBride]

I found a temporary workaround and shared it with ESRI today. Since I’ve been trying to achieve the same thing for a while, I wanted to share it here as well.

After upgrading to Monitor 2024.1, I discovered the service-check feature, which works as expected. It properly logs 500 errors and can even monitor a specific feature within a web map to confirm the service is truly working. This helps avoid false positives where AGS reports a service as running, but no geometry appears in the web map. Normally, we’d restart the service to resolve this, but the goal is to catch the issue before the customer notices. The service-check logs confirm spatial data is drawing correctly. However, the current limitation is that notifications don’t trigger when errors occur (e.g., a 500 error). Instead, Monitor only alerts on metric changes, not the result of those metrics.

As a temporary solution, I’m using Collections to monitor whether a service is Started or Stopped. Collections allow monitoring of System State, which reflects service status. I created multiple collections based on projects with multiple services. In Expression Builder, I select all relevant services and set the System State to STOPPED (or STARTED). Then, I configure notifications to alert based on Updates.

In short:

1. Create a collection based on services that are up/down.
2. Set it to check every minute.
3. Enable notifications for changes in collection status.
4. Rename the notification email to “ArcGIS Monitor Service STOPPED” to include a table with the affected service name.

This method ensures alerts trigger when services start or stop, providing proactive monitoring