topic Re: Performance Degraded - Server - Shifting Directories on Network Storage in ArcGIS Enterprise Questions

Performance Degraded - Server - Shifting Directories on Network Storage

User35488 — Tue, 08 Jan 2019 12:34:30 GMT

Hello There,

We are facing drastic performance issue post shifting the directories(cache, jobs, output, system & conf) on Network storage from local server.
Kindly help me to troubleshoot this issue.
Which areas i shall be diagnosing to resolve this issue

Environment:

ArcGIS Server 10.6
Windows 2012 R2 Server

Thanks for kind support
-AS

Re: Performance Degraded - Server - Shifting Directories on Network Storage

George_Thompson — Tue, 08 Jan 2019 13:08:34 GMT

There are a lot of considerations for this issue.

- What type of network storage is being used (NAS, SAN, etc.)?

Choosing a NAS device for ArcGIS Server—ArcGIS Server Administration (Windows) | ArcGIS Enterprise

- Is the network storage in the same location as the server machines?

- Was there a reason for moving the config stores form the local machine to the network?

I have seen in the past where if there is a network hick-up to the config stores that the site may become unresponsive. I have also seen where issues with I/O on the network can cause issues.

Re: Performance Degraded - Server - Shifting Directories on Network Storage

RandallWilliams — Tue, 08 Jan 2019 17:13:45 GMT

As George Thompson‌ mentioned, there are a lot of potential reasons for this. I'd also add that there can be latency related to Active Directory performance (EG, the domain controller that governs the location that this share is located, DNS latency, etc. Have your network run a wireshark trace and check for delays.

Re: Performance Degraded - Server - Shifting Directories on Network Storage

JonathanQuinn — Tue, 08 Jan 2019 18:16:23 GMT

Echoing Randall Williams‌ and George Thompson‌, latency problems can be tricky to troubleshoot outside of your IT staff. I've found that IoZone is a useful tool that you can use to take ArcGIS Server out of the picture. It simulates read/write operations against a directory and measures the throughput in kb/second. You can run it against your network share and then your local directory to at least give you an idea of what your network share throughput needs to be at to perform as well as your local directory:

On the ArcGIS Server machine, install the following software:

IOZone (http://www.iozone.org/src/current/IozoneSetup.exe)
cygwin (http://cygwin.com/setup-x86_64.exe)

*Note, there have been issues running IOZone with paths to the network from the machine the folder is shared to, meaning running IOZone from m1 to a share like \\m1\<path>

"C:\Program Files (x86)\Benchmarks\Iozone3_414\iozone.exe" -Ra -f //networkpath/folder/file -b %USERPROFILE%\Desktop\networktest.xls

* Ideally, the //networkpath/file parameter would be set to the share to your server directories, for example //<machine>/arcgisserver/directories/file. In this directory, it'll create a simple file called "file" and run it's tests on that file.

Run the following command as well when you're logged in with the ArcGIS Server account on that machine:

"C:\Program Files (x86)\Benchmarks\Iozone3_414\iozone.exe" -Ra -f C:\arcgisserver\directories\file -b %USERPROFILE%\Desktop\localtest.xls

The parameters listed above are defined below:

-R = generates an Excel file

-a = uses full, automatic mode, meaning tests files with records sizes from 4k to 16M for file sizes 64K and 512M

-f = sets file path for output

-b = sets file path for Excel file.

If you're interested in knowing more about the tests that it will run, they can be read up on through the following pdf.

IoZone documentation:

http://www.iozone.org/docs/IOzone_msword_98.pdf

Re: Performance Degraded - Server - Shifting Directories on Network Storage

User35488 — Wed, 09 Jan 2019 08:39:54 GMT

Thanks a lot for excellent inputsGeorge Thompson , Jonathan Quinn ,Randall Williams.

We basically moved the folders on NAS to be accessible by clustered server / New Site.

ArcGIS Server and NAS are on different locations and deployed on different domains

I will try the suggestions one by one to find out the bottleneck. Thanks again

-AS