ArcGIS Monitor 2023.3.1: High CPU utilization

[AyushAggarwal]

We have several ArcGIS Enterprise Deployments and a separate ArcGIS Monitor deployed for each environment. The monitor is deployed on a dedicated server with 2 core CPU and 16 GB RAM for most of them, along with Monitor Agents installed separately on each Enterprise component server. 

Recently, we've started noticing a behavior on some of the environments where the CPU of the monitor machine shoots upto a 100% and renders it unable to browse the monitor url. 

We've tried restarting the Monitor service, also rebooting the entire machine, however the CPU is still at capacity but as soon as the monitor service is stopped, the CPU drops to about 20-30%. 

This issue was first noticed on a monitor agent machine where we ultimately had to uninstall it and then register the machine as a host. After this we've had 2 more occurrences on the Monitor server itself at random. 

We're yet to figure out a solution or what might've caused this since we haven't made any changes apart from the automatic windows updates. Is there an issue with any specific windows update recently?

We will try to stop the windows defender service on these problematic severs and see if this could be temporary solution. Any other suggestions would be helpful.

Thanks

[StevenB]

Generally, when we provide initial sizing requirements for an ArcGIS Monitor deployment we always start off with a 4-core/16GB RAM machine and specify that vertical scaling may be required depending on the number of users accessing monitors interface and the number of components ArcGIS Monitor is collecting metrics for.

Keep in mind if you're in a virtualized environment your current 2-cores may only be a single core if hyperthreading is enabled. Are your separate environments, separate "production environments" or separate staging and/or development environments? Not sure how big these other environments are but you potentially could get better utilization of your host resources running a single "larger" ArcGIS Monitor machine collecting metrics for all your environments and organizing them into collections. 

I think a lot users get thrown off with ArcGIS Monitor's system requirements info for hardware.

These stated specs are for the ArcGIS Monitor server or agent processes only and do not include Windows Server OS and if installed on the same host the PostgreSQL processes etc. 

I once had a client who deployed a 2-core/4 GB RAM machine because that is what the system requirements stated, which didn't run well until they updated to our 4-core/16 GB RAM recommendation. 

For ArcGIS Monitor Server, a minimum of 2 CPU cores and 4 GB of RAM are required.

For ArcGIS Monitor Agent, a minimum of 1 CPU core and 2 GB of RAM are required.

Note: Depending on your organization's needs, and performance and capacity expectations, the implementation may require more cores or RAM.

---

[AndrewSakowicz]

I recommend upgrading to 2024.0.1 and provide the below min requirements.

Note, the local light agent is a new implementation.

ArcGIS Monitor system requirements—ArcGIS Monitor | Documentation

[AYUSHYADAV]

Hi @AndrewSakowicz ,

In the documentation, it states that ArcGIS Monitor requires 2 CPU cores, the ArcGIS Monitor agent requires 1 CPU cores, and the database requires 2 CPU cores. If I want to install all three on the same machine, will 2 CPU cores be sufficient, or will the machine require a total of 2 + 2 + 1 = 5+ core CPUs i.e 8 core CPUs?

Thanks

Ayush

[AndrewSakowicz]

4 cores should be sufficient.  But keep an eye on the PostgreSQL.  It can grow if there are many users and / or complex analysis. 

[Shawn_NG]

@AyushAggarwal I have deployed 2023.3.1 version on a server with 2 core (4 logical processors) and 16GB RAM in my organization and haven't faced any CPU related issues yet. My guess would be to troubleshoot the issue you should start with Event Viewer (Application and System) and check what entries are being written. In your screenshot Event Viewer is the second application with high CPU suggesting that a lot of activity is going on there.

Few items to keep in mind/ questions I would ask:

• Since your deployment is on the C drive, ensure to have enough space to support both Monitor and PostgreSQL application.
• Keep a watch on the top CPU consuming items in the list. Do they change or remain constant, this might help in getting to a conclusion.
• Does the CPU peak to 100% as soon as Monitor service starts or does it take some time.
• Did the issue start at the same time when Monitor was first installed or did something trigger this.

[AyushAggarwal]

Hi @Shawn_NG,

The top CPU consuming processes were windows defender and site24x7. The CPU shoots up 100% as soon as we start the ArcGIS Monitor service, and it stays at capacity constantly. We started facing this issue out of the blue even though we did not make any architectural changes.

We did and hit and trial and was able to fix the issue by changing the service account for monitor service (Arcgis_Service) to another domain account as well as to local system account. However, there aren't any difference in new the service account settings with that Arcgis_Service. We have other Enterprise components running from Arcgis_Service account which are on another machine but in the same domain. There we do not see any CPU Spikes. 

[AyushAggarwal]

Update:
We got this issue again on a different environment. This time the issue was resolved without having to change the service account for the monitor agent and server. 

We just uninstalled > cleaned all the directories for residual monitor files (Within Program files and App data under user profile) > Reinstall and reregister monitor agent. 

We saw significant improvement in CPU management for the Monitor agent machine once this was done. 
We have not yet tested ArcGIS Monitor 2024. Hopefully this issue is not propagated there.