ArcGIS Monitor 2023.2 - ArcGIS server services stats disapeared over the weekend

[WalterSimonazzi_VicPol]

Hi all,

For some unknown reason, our instance of ArcGIS Monitor 2023.2 has lost all stats of ArcGIS Server services over the weekend. Last week we had a consumer of one of our gpservices bombarding an ArcGIS Server gpservice with requests. ArcGIS Monitor was a fundamental tool for us to prove that the number of requests from this service went from 3 requests/s to almost 18req/s which obviously put the service under stress and cause a big decrease in response time. (graph below)

Today Monday, we went to see how the service behaved during the weekend and found that there are not stats, any stats. Monitor started again to gather stats of this service at around 7AM today – we don’t know why it stopped nor why it started tp gather stats today again at 7AM….

Last 7 days stats: as you can see we don’t have any data for the past 7 days….

We have also found that all the other services in ArcGIS Server registered in Monitor showed the same issue with a sligh difference,  they started to collect stats again at a different day/time. See one example below.

Can someone tell us what’s going on and how to fix this issue? And how to retrieve all the past stats.

@derekl 

[DerekLaw]

Hi @WalterSimonazzi_VicPol,

> For some unknown reason, our instance of ArcGIS Monitor 2023.2 has lost all stats of ArcGIS Server services over the weekend. 

I'm sorry to hear about this odd behavior. This is unusual. Can you please open an issue with Esri Tech Support so they can help investigate further? 

• What release of ArcGIS Enterprise are you running?
• When Monitor stopped collecting metrics, did anything happen to the system? network? maybe some sort of security scan?
• You mentioned for other services - Monitor started re-collecting metrics at different times - do they have the same or different recycle settings? (you can check in ArcGIS Server Manager > Services > Manage Services > select a service > Processes)
• Do you have any other ArcGIS Server sites registered with Monitor? Did the same thing happen, or did this only happen to the single ArcGIS Server site?

Again, please work with Esri Tech Support - as I don't think we'll be able to troubleshoot on the forums.

Hope this helps,

[WalterSimonazzi_VicPol]

Thanks Derek.

We are on 10.8, althought currently migrating to 11.2.

I have seen there's an "data retention policy" that controls how long metrics and logs are retained. In our case we have 180 days, which is roughly the time we've been using this version of Monitor....

How this retention police work? does it deletes all logs and metrics when 180days are reached? 

[DerekLaw]

Hi @WalterSimonazzi_VicPol,

> How this retention policy work? does it deletes all logs and metrics when 180 days are reached? 

The default data retention policy for the ArcGIS Monitor database is 180 days, which means collected data metrics will be retained for 6 months. Data collected beyond this date (i.e., older than 6 months) will be deleted from the database. 

I don't believe this is the cause of the behavior you're seeing.

Hope this helps,

[WalterSimonazzi_VicPol]

You are right, that's the not issue. I see that it's not happening to all services, just a few of them in particular geoprocessing services.

The issue started to happen after a period where the gp service.