AnsweredAssumed Answered

ArcGIS 10.4.1 to 10.5 Upgrade Failed: REST handler inaccessible...

Question asked by knoop_umich on Feb 21, 2017
Latest reply on May 7, 2018 by NicolasGIS

This is less of a question, and more to get something surfaced for other's Google searches, if they are experiencing similar symptoms...

We recently attempted an upgrade from ArcGIS for Server 10.4.1 to ArcGIS Enterprise 10.5 on Windows. The instance also included IIS Web Adapters. The upgrade went smoothly until we reached step 6 in Upgrade ArcGIS Server, where one clicks on "Continue Server Upgrade" on the ArcGIS Server Manager webpage. After working for a bit, an error is returned:
Upgrade step 2 of 2 : Failed. REST handler inaccessible when attempting to upgrade
The ArcGIS server log also shows a bunch of "REST inaccessible" warnings. 
We tried various reboots, repairing the 10.5 installs, uninstalls/reinstalls, and kept getting stuck at this step 2 of 2. Eventually we are able to complete this step, after we shutdown the IIS service completely, and tried one more time. 
At this point it appeared that the update step completed successfully, however, we were unable to register the new 10.5 Web Adapters, and we noticed more REST inaccessible warnings in the server log, such as:
<Msg time="2017-02-15T11:57:18,957" type="WARNING" code="7712" source="Server" process="4144" thread="1" methodName="" machine="xxx" user="" elapsed="">An error was encountered while synchronizing with the config store. REST handler inaccessible when attempting to synchronize</Msg>
After further investigation with Esri's help, we determined that server was not listening on port 6443, as it should've been. It was only listening on 6080. We verified this with:
netstat -ao | findstr "LISTENING" | findstr "6080 6443"
To get the application listening again on port 6443 we had to complete the following steps:
  1. Log in to the admin page at http://localhost:6080/arcgis/admin 
  2. Navigate to security/config/update
  3. Change the Protocol drop-down to HTTP Only
  4. Click Update to commit the change
    1. The web server (service) will restart.  Wait for that to complete.
  5. Change the Protocol drop-down back to HTTP and HTTPS
  6. Click Update to commit the change
    1. The web server (service) will restart.  Wait for that to complete.
  7. Using the Services.msc, restart the arcgisserver service
  8. Re-check to see if the server is again listening on 6443 by running the net stat command above.  In our case, it was.
No idea what went wrong during the upgrade to cause this disconnect on 6443, however, we are finally out the other side and successfully running 10.5. Prior to this we had upgraded three other instances form 10.4.1 to 10.5 and encountered no issues at all. The only difference being that this last system supports many more web services (~200), so perhaps a time-out somewhere along the way, given how long it takes to start-up all those services, screwed things up?
-peter

Outcomes