Performance Issues with Redundant ArcGIS Servers behind F5 Load Balancer

Question

Hi Esri Community,

I'm seeking assistance with a performance issue we've encountered with our ArcGIS Server setup. Our environment consists of redundant ESRI ArcGIS map servers with web adapters, all configured behind an F5 BIG-IP load balancer. We're running ArcGIS Server Enterprise Version 10.9.1 with all the latest patches, and F5 BIG-IP Version 15.1.8.2 Build 0.0.17.

When both servers are active in the load balancer, we experience significant performance degradation. Here are some of the specific issues:

* ArcGIS Server Manager Loading Time: It takes over 5 minutes for the ArcGIS Server Manager to load. After logging in, it takes several more minutes for the pages to fully render.
* Publishing Feature Services: What normally takes 1-2 minutes now takes anywhere from 20 to 40 minutes.
* ArcGIS Desktop Connection: Connecting to the ArcGIS Server through ArcGIS Desktop is extremely slow.
* Managing Services: Starting or stopping services is very slow and often results in errors.

Troubleshooting Steps Taken:

Scenario 1: Both Servers Active

* We ran tests with both servers active in the load balancer.
* Result: ArcGIS Server Manager took over 5 minutes to load. Significant delays were observed in all operations, including publishing services, connecting through ArcGIS Desktop, and managing services. Most attempts to publish services resulted in errors.

Scenario 2: Disabled Server 2

* We disabled Server 2 in the load balancer and shut it down.
* Tested the system with only Server 1 active.
* Result: Same slow performance as scenario 1 however publishing and managing services were possible all be it slow.  Server 2 had antivirus software enabled, which we removed from Server 1 for testing.

Scenario 3: Disabled Server 1

* We disabled Server 1 in the load balancer and shut it down.
* Tested the system with only Server 2 active.
* Result: Same slow performance as scenario 1 however publishing and managing services were possible all be it slow. Server 2 still had antivirus enabled, but we concluded antivirus was not the issue.

Additional Information:

* Antivirus software was ruled out as a contributing factor after testing.
* The primary ArcGIS Site Admin account has full control of all Esri directories.
* Feature layer data source comes from a file geodatabase stored on a shared data storage server separate from the two GIS servers.
* The third-party software, that consumes the GIS services through their custom API, requires the data source to be a file geodatabase and does not support enterprise geodatabases.

We are looking for any insights or suggestions from the community on how to resolve these performance issues. Has anyone experienced similar problems with redundant ArcGIS servers behind an F5 load balancer? Any configuration tips or troubleshooting advice would be greatly appreciated.

Thank you in advance for your help!

np-gis · Answer

This is somewhat anecdotal and late, however I just came across this question. My old organization had similar issues with F5 and specifically ASM.  Once ASM was disabled for all traffic to the ArcGIS enterprise deployment, all the performance issues went away. Reenabled, they came back. I don't know the root cause as the firewall team opted to just leave it disabled and not fine tune it.

MattMoore · Answer

@NicoleCeranek_MCTX did you ever find a solution to this? Seeing similar performance in a deployment currently.

abdelsalmelmoursi · Answer

Hello NicoleCeranek_MCTX,

As per your post, it seems you are facing some performance issues with your ArcGIS Enterprise setup. Here are a few additional questions we can explore to help troubleshoot the problem:

* Server Resource Utilization: Have you checked the utilization of RAM and CPU on your ArcGIS Server machines? It's important to ensure that the server hardware specifications are adequate to handle the workload. If the CPU or memory utilization is consistently high, you may need to consider upgrading the server hardware.

* ArcGIS Server Service Account: Does the administrative account used to operate the ArcGIS Server service have full control over the shared storage where your database is stored? Ensuring proper permissions for the service account is crucial for smooth operation.

* ArcGIS Server Configuration Storage: Where are the ArcGIS Server directories and configuration files stored? Are they located on a local machine or on a network-shared folder? Storing these files on a network-shared folder can sometimes cause performance issues like network issues and we need to make sure the ArcGis service Account have full control on that folder

* Antivirus Configuration: Have you tried disabling the antivirus software temporarily to see if it's affecting the performance of your ArcGIS Enterprise setup? Certain antivirus configurations can interfere with the normal operation of ArcGIS services.

* Windows Event Logs: Checking the Windows Server event logs on both the ArcGIS Server and the load balancer machines can help identify any abnormal activities or errors that may be contributing to the performance issues.