Slow to publish services after adding a second machine to site

[TracyArchibald]

We recently added a second machine to our arcgis server site.  We did this mostly for redundancy, rather than performance reasons.

After adding the machine, we've had some issues with it being slow to publish, administer and view services in Manager.  Using them from the rest endpoint seems fine.

It's so slow I'm concerned about a revolt, but with less than 5 people doing the things that are slow, I do have a bit of wiggle room.

It was fine before adding the second machine.

Environment:

10.4.1 - fully patched  (including both publishing patches)

2 blade servers - different versions of windows (I know - bad!)  I plan to upgrade the older one after summer, if I can get that much time.

1 web adapter in a VM

Config store is on a network file share

[JonathanQuinn]

Was it slow after moving the directories to the file share or only after adding the second machine to the site?  Are you using separate clusters?  We recommend only one cluster for performance and stability reasons:

About single cluster mode—Documentation (10.4) | ArcGIS Enterprise 

[TracyArchibald]

One cluster, two machines.

We moved it to the file share almost a year ago when I tried the first time to add a second machine.  It was a horribly under powered VM so we pulled it out.  But the performance with the file share has been fine since then. 

And like I mentioned, using the services are fine, but "administering" them are not.

The setting mentioned in in the admin console is currently set to "false".

[JonathanQuinn]

I suggest setting that to True and seeing what kind of performance improvement you get, if any.  It's likely there's enough latency between the machines during the internal communication to cause problems.

[JayantaPoddar]

Check if firewall is allowing Machine 2 to read the config-store and directories. Ensure that the following are allowed through firewall.

1. Ports used by ArcGIS Server—Installation Guides (10.4.1) | ArcGIS Enterprise esp. 4000-4002, 4004

2. Following programs

• <ArcGIS for Server installation location>\bin\ArcSOC.exe
• <ArcGIS for Server installation location>\framework\etc\service\bin\ArcGISServer.exe
• <ArcGIS for Server installation location>\framework\runtime\jre\bin\javaw.exe
• <ArcGIS for Server installation location>\framework\runtime\jre\bin\rmid.exe

Fiddler might help you narrow down to the cause of the issue.

Download Fiddler Web Debugging Tool for Free by Telerik 

[TracyArchibald]

I've (temporarily) opened up the firewall to network traffic and it *seems* better.  I'll get my team to test it tomorrow.  Then I'll set it up correctly.