Slow performance administering services in ArcCatalog and ArcGIS Server Manager 10.1

[BrettElliot]

Hi everyone,

Recently we have been having issues with access the map services in ArcCatalog.  After trying to recreate the error on a test machine, we narrowed down part of it to having the Configuration Store set to a network drive as opposed to C:/.

Originally, we had set the configuration store to a network drive because "the location must be accessible to all machines that are registered to the site".

We shared the C:\arcgisserver\config-store across the network so that all machines could access it instead.

Since changing this setting, the response time is much much faster than it was before.  It used to take ~30 seconds to pull up the 'right click' menu in ArcCatalog, or view the services directory in ArcGIS Server Manager.

If anyone else is having similar issues, I hope this can help.

Any insight to why this would cause everything to slow down would also be appreciated.

Thanks,

[RoyceSimpson]

Hello Alexandra.  This has been a thorn in our side, off and on, for a long time.  Here is where we are now with all this:  Our shared storage system that our machines within our AGS site use (cache, input, output, config-store, etc... folders) has been a source of huge performance issues.  One thing that helped out a lot was to turn "oplocks" (opportunistic locking) off for our AGS share.  A second thing that came up that was showing huge performance issues was the location of our "output" folder.  It was in the same share as our other folders.  I moved the output folder to one of the c drives on one of the machines in the site and shared that folder with the arcgis admin account, and now, performance is 10+ times faster.

Hope this helps.  Let me know if you need any more explanation of the above work-arounds.

-Royce

[JonathanQuinn]

These problems typically come down to network latency and inter-machine communication.  You'll need to make sure that your shared storage can perform well with random read and write operations:

Choosing a NAS device for ArcGIS Server—ArcGIS Server Administration (Windows) | ArcGIS Enterprise 

A good tool for this is IOZone which can capture throughput of reading and writing to a certain directory:

On the ArcGIS Server machine, install the following software:

1. IOZone (http://www.iozone.org/src/current/IozoneSetup.exe)
2. cygwin (http://cygwin.com/setup-x86_64.exe)

*Note, there have been issues running IOZone with paths to the network from the machine the folder is shared to, meaning running IOZone from m1 to a share like \\m1\<path>

Sign in to the Server machine as the ArcGIS Server account and run the following command from the command line:

• "C:\Program Files (x86)\Benchmarks\Iozone3_414\iozone.exe" -Ra -f //networkpath/folder/file -b %USERPROFILE%\Desktop\networktest.xls

* Ideally, the //networkpath/file parameter would be set to the share to your server directories, for example   //<machine>/arcgisserver/directories/file.  In this directory, it'll create a simple file called "file" and run it's tests on that file.

If you have seen no problems with a server configured with local directories, run the following command as well when you're logged in with the ArcGIS Server account on that machine:

• "C:\Program Files (x86)\Benchmarks\Iozone3_414\iozone.exe" -Ra -f C:\arcgisserver\directories\file -b %USERPROFILE%\Desktop\localtest.xls

The parameters listed above are defined below:

-R = generates an Excel file

-a = uses full, automatic mode, meaning tests files with records sizes from 4k to 16M for file sizes 64K and 512M

-f = sets file path for output

-b = sets file path for Excel file.

If you're interested in knowing more about the tests that it will run, they can be read up on through the following pdf.

IoZone documentation:

http://www.iozone.org/docs/IOzone_msword_98.pdf

By taking ArcGIS Server out of the picture and comparing the results of simply reading and writing files to your shared location, you can determine if there's a networking problem to address.

The other thing to look at is if your site is configured to use a single cluster and if it's in single cluster mode.

About single cluster mode—ArcGIS Server Administration (Windows) | ArcGIS Enterprise 

If this is an upgraded environment, you may have multiple clusters or single cluster mode may be disabled, which enables the communication between the machines that affects performance.

[Anonymous User]

Thanks for the tips! I'll talk to the infrastructure guy that set up the shared NAS and see if I can uncover anything there. I find it interesting that performance is just fine using the NAS with a single machine... the performance problems kick in when 2+ machines are in use. Also, I did confirm that single cluster mode is enabled.

[Anonymous User]

Royce Simpson‌'s suggestion to disable oplocks in our NAS solved the problem! I also had a premium support case open where Patrick from Esri suggested the same thing. He said "[oplocks] is known to cause some issues with the GIS Server software and is commonly used in NAS to speed up performance. For the Enterprise ArcGIS Software it actually kills performance and it gets worse with multiple machines accessing the shared content."

Thank you Royce and Patrick!

[RolandsRihters]

We had thesame issue with 10.3.1.

Opening ports 4000-4004 resolved the issue