ArcGIS Monitor 2023 Postgres Drive Fills Quickly

[AlexBakhtin]

Hello,

We've been using ArcGIS Monitor 2023 since February when it was released and we have so many alerts that go off daily that our 100GB drive that has the postgres db is almost full. For reference, we currently have ~1500 components that are registered to Monitor and we are expecting that number to increase in the coming months. I recall ArcGIS Monitor automatically clears out old alerts after a certain time but don't remember how often. Would it be possible to change the frequency of how often it happens? Or what would the manual process look like if we were to create a procedure to do it ourselves?

Thanks!

Alex Bakhtin

[AndrewSakowicz]

@AlexBakhtin   Thank you for this feedback.

As far as high level storage estimates, for 5 min resolution, please budget 100 GB per 1000 components for 12 month retention.  In the 2023.2 (planned for June 2023), we'll provide UI to change this default to lower.  This will allow users to free up space.

Can you provide more info regarding the type of components and collection interval?  Are these mainly arcgis server services, collecting 5 min?  How many arcgis server node?  They all would collect process information, which might explain the storage usage.  We'll update documentation shortly.

[MKR]

Please let me know once documentation is updated.

We are monitoring approx. 5000 components (different environments) and currently at around 330gb for - would be good to understand what storage requirements will be going forward. 

We have left the collecting time as default for all components. Is there a way to change this setting for all components or is this an individual setting per component?

For retention period for collection metrics etc.. is this setting controlled in Administrator under the Log Retention Period option or is this just the settings for the log files for Monitor itself?

[AndrewSakowicz]

@MKR , Of course, we'll announce shortly once the doc is updated. 

The UI for partition retention will be under Administration>Database.  This feature is scheduled to be part of 2023.2, June 2023. 

So please allocate disk space for the next 2-3 months.  In your case, it might be additional 300-400 GB. 

If provisioning this disk space is an issue, please let us know.  We can walk you through how to do this manually: this feature frees up disk space by dropping PostgreSQL older partitions in metrics_data table.  

[MKR]

Thanks Andrew,

We should be ok with disk space. Likely we would only want to keep 2-3 months worth of historic data at any one time.

[AmaiaMartinez]

Hi there!

We are monitoring approx. 2000 components. We are experiencing the same issue... C drive is full again and again...

We cannot continue extending it, so we'd need you to provide how to do this manually (to free up disk space by dropping PostgreSQL older partitions in metrics_data table).  

Additionally, we have found that suddenly there is a big peak on the disk space usage (see the pattern):

Thank you in advance!

[AndrewSakowicz]

I will contact you directly.

[AlexBakhtin]

Hi @AndrewSakowicz, thanks for the information! With regards to the components and collection interval:

• You are correct, these are mainly ArcGIS server services that collect every 5 minutes.
•  We are monitoring seven ArcGIS Server nodes as of now but only one of them is the "large one" that contains the bulk of our services (~1100) and the other 400 services are spread across 6 different nodes. We will also be cleaning up 
• We doubled the disk space this morning and will keep it at 200GB per the recommendation for 100GB per 1000 components and scale it up in the future but ultimately we will need to adjust the retention period

[AlexanderSchiestl]

Hi Andrew,

we are also facing the issue that ArcgisMonitor postgresql DB won't shrink.
We tried to reduce the log time to 1 day, to unregister various components and even a "full vacuum" via pgAadmin didn't reduce the size of the database.

In our hosting environment, it is necessary to manage the processes related to monitored items and growth rates on the DB side.

Can you please provide us with instructions on how to manually clean up the ArcgisMonitor entries of the postresql DB?

We Use MS Server 2022 (latest patches), ArcGIS Monitor 2023.1 (latest) and pgAdmin 6.19.

Thank you very much.

Alexander

[AndrewSakowicz]

@AlexanderSchiestl please follow:

1. back up your pg database, https://www.postgresql.org/docs/current/backup.html
2. Drop old partitions in metrics_data table