ArcGIS Server Site (10.9.1) Running Very Slow

[Sunnywaygis]

Hi All, We are experiencing an odd issue within our ArcGIS Enterprise environment (ver 10.9.1). We are encountering Error (ERROR 001369) during publishing. All steps for publishing (from desktop) take much longer than expected. These are few observations: 

• Launching, Statistics page from ArcGIS Manager > Logs > Statistics takes very long time and fails.
• Published map services are performing well. User are able to access content with optimal performance, no impact there.   
• Manager logs don’t show any error that indicates the reason for slowness. 
• Accessing config store from GIS server machines is quick, using windows explorer
• GIS server resources are optimal, less than 15% usage.  

This site is an active-active ArcGIS server site that is federated with Portal. Our configuration is as follows:

• Two GIS servers with a common network share.
• One web adaptor with each GIS server, and a load balancer for traffic

We are trying to find  where the bottleneck is coming from and if there are any tool/options that can help in identifying the issue by checking end to end communication and capturing time for each step. 

Thanks in advance

JS

[A_Wyn_Jones]

Has the network share been configured for any kind of redundancy? Often see similar issues when customers configure a DFS for a network share which contains the ArcGIS Server directories.

Please see: https://enterprise.arcgis.com/en/server/latest/install/windows/choosing-a-nas-device.htm#:~:text=dat....

[Dan_Brumm]

I know this doesnt help but we have had a similiar issue for years.Specifically dealing with CSV's being published directly through the Portal front end. I would 90% of the time it fails with that same error code. We never really found a fix. We developed a work around instead where we publish through ArcGIS Pro. 

We did work with esri on this  for months. I want to say at least 6 months worth of troubelshooting and we could never find a root cause. 

[ArchitSrivastava]

Hello @Sunnywaygis ,

It seems like somewhere there is a network bottlenecks as you expect. However, It would surely not be an easy thing to figure out.

It seems that ArcGIS Server has issues in accessing components on the same machines as in this scenario the statistics and logs would be stored individually on their own as opposed to config-store and directories being on shared directory. This could be causing a delay in response

I would suggest taking the following approach.

Cover all the bases

• Run the "Configure ArcGIS Server Account utility" on both the machines and make sure you point to shared location and account used is same
• Check the access for ArcGIS Server Service account on "C:\Python27" and "C:\Program Files\ArcGIS\Server" on both ArcGIS Server machine
• Make sure there are exceptions added in the antivirus running on the machines running ArcGIS Server, refer to "ArcGIS Antivirus Guidance"
• Make sure all the ports required are working, refer to "Ports used by ArcGIS Server 
• Check if there are any "oplocks" are being possibly generated on the Shared file server
• Make sure all the Required ESRI Patches are installed on both the ArcGIS Server machines

If you perform the above and it does help (make sure you the above first)

A bit more advanced ways

• Finding which step fails
  • While publishing it takes two steps mainly "Packaging " and "Publishing"
  • Change the logging level for "Publishing Tool on both the server Enabling info-level geoprocessing messages : Troubleshooting geoprocessing REST services—ArcGIS Server | Documentation for ArcGIS Enterprise
  • Once, you enable this run the tool again and navigate to "ArcGIS Server Manager" > Site > Jobs > Under Service select the GP service and query. This will show you the Job ID generated for the latest run of the tool. Once you open that, it will show you details of the JOB run.
  • Check the logs for details of which step fails
• Publish a service to each server separately and see if any one of that fails
  • If yes, which server is it and check the Network connectivity on that
  • If not, Load balancer configuration could be at fault
• Where is that Data coming from and could that be a cause, depending on which one fails/succeeds further steps can be decided
  • Publish a test map service to ArcGIS Server and allow the data to be copied to Server, does that fail?
  • Publish the data from Database registered to ArcGIS Server and does that fail as well.
•  Configure Fiddler on the Desktop machine and capture the full workflow of what steps are being taken, this will help in the following:
  • If or any steps are being missed
  • Which exact step is taking the maximum time

I would say that this would be a good starting point. Please be aware that there are just ways to narrow down the issue, either of these could/could not be a solution but this will help us in understand where the problem resides for sure.

Let's try this and let me know, how it goes and then we take it further.

Hope it helps!

-Archit