Performance Degraded - Server - Shifting Directories on Network Storage

Question

Hello There,

We are facing drastic performance issue post shifting the directories(cache, jobs, output, system & conf) on Network storage from local server.
Kindly help me to troubleshoot this issue. 
Which areas i shall be diagnosing to resolve this issue

Environment:

ArcGIS Server 10.6
Windows 2012 R2 Server

Thanks for kind support
-AS

User35488 · Answer

Thanks a lot for excellent inputsGeorge Thompson , Jonathan Quinn ,Randall Williams.

We basically moved the folders on NAS to be accessible by clustered server / New Site.

ArcGIS Server and NAS are on different locations and deployed on different domains

I will try the suggestions one by one to find out the bottleneck. Thanks again

-AS

JonathanQuinn · Answer

Echoing Randall Williams‌ and George Thompson‌, latency problems can be tricky to troubleshoot outside of your IT staff. I've found that IoZone is a useful tool that you can use to take ArcGIS Server out of the picture. It simulates read/write operations against a directory and measures the throughput in kb/second. You can run it against your network share and then your local directory to at least give you an idea of what your network share throughput needs to be at to perform as well as your local directory: On the ArcGIS Server machine, install the following software: * IOZone (http://www.iozone.org/src/current/IozoneSetup.exe) * cygwin (http://cygwin.com/setup-x86_64.exe) *Note, there have been issues running IOZone with paths to the network from the machine the folder is shared to, meaning running IOZone from m1 to a share like \m1\ Sign in to the Server machine as the ArcGIS Server account and run the following command from the command line: * "C:\Program Files (x86)\Benchmarks\Iozone3_414\iozone.exe" -Ra -f //networkpath/folder/file -b %USERPROFILE%\Desktop etworktest.xls * Ideally, the //networkpath/file parameter would be set to the share to your server directories, for example ///arcgisserver/directories/file. In this directory, it'll create a simple file called "file" and run it's tests on that file. Run the following command as well when you're logged in with the ArcGIS Server account on that machine: * "C:\Program Files (x86)\Benchmarks\Iozone3_414\iozone.exe" -Ra -f C:\arcgisserver\directories\file -b %USERPROFILE%\Desktop\localtest.xls The parameters listed above are defined below: -R = generates an Excel file -a = uses full, automatic mode, meaning tests files with records sizes from 4k to 16M for file sizes 64K and 512M -f = sets file path for output -b = sets file path for Excel file. If you're interested in knowing more about the tests that it will run, they can be read up on through the following pdf. IoZone documentation: http://www.iozone.org/docs/IOzone_msword_98.pdf

RandallWilliams · Answer

As George Thompson‌ mentioned, there are a lot of potential reasons for this. I'd also add that there can be latency related to Active Directory performance (EG, the domain controller that governs the location that this share is located, DNS latency, etc. Have your network run a wireshark trace and check for delays.