ArcGIS Server 10.4.1 to 10.5.1 Failed Upgrade and Solution

RyanMartin2 · ‎02-21-2018

This will not be relevant to everyone but I am sharing in case it helps someone else overcome the difficulties I experienced.

We recently attempted an upgrade of ArcGIS Server from 10.4.1 to 10.5.1 on one of our test/R&D servers running Windows Server 2008 Standard SP2 but we encountered difficulties. Specifically this error message at stage 2 of 2 of the upgrade process:

Upgrade step 2 of 2 : Failed. Refresh auto-deployed services during upgrade

If you run the task Asynchronous you can use the ArcGIS Server back end for the duration the upgrade task is processing and when it fails it locks you out again. In the server logs for the ArcGIS Server back end, which I accessed while the Asynchronous task was running, I discovered this error repeated numerous times with type "SEVERE" and code "6563":

"Failed to create the service System/PublishingToolsEx.GPServer"

If you look at the Asynchronous task list and select the upgrade job it looked like this (effectively a list of services starting and restarting continuously due to errors with the publishing tools service, then it eventually fails):

Upgrade step 2 of 2 : Started.
Upgrade step 2 of 2 : License check started.
Checking and validating server licenses.
Upgrade step 2 of 2 : License check completed.
Upgrade step 2 of 2 : Configuring application server.
Starting server on machine '{SERVERDOMAIN}'.
Upgrade step 2 of 2 : Configured application server.
Upgrade step 2 of 2 : Configured platform services.
Upgrade step 2 of 2 : Updating auto-deployed services.
Stopping the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsManagement' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsManagement' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsManagement' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsManagement' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsManagement' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsManagement' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsManagement' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsManagement' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsManagement' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Starting the service 'CachingControllers' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsManagement' on machine '{SERVERDOMAIN}'.
Stopping the service 'GeoAnalyticsTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'ReportingTools' on machine '{SERVERDOMAIN}'.
Stopping the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Starting the service 'PublishingTools' on machine '{SERVERDOMAIN}'.
Upgrade step 2 of 2 : Failed. Refresh auto-deployed services during upgrade

I tried a number of obvious things including restarting the services and restarting the server to no avail. After some online research I got to the root cause of the error with troubleshooting advice from foreign language forums. The upgrade only works with IE9 or newer installed on the host machine.

We were running Windows Server 2008 Standard SP2 (so not Server 2008 R2) and its default browser was IE7. Apparently the upgrade process for ArcGIS Server 10.5.1 uses the browser on the host machine to run upgrade scripts even if it doesn't tell you as much in the error messages.

To be fair the system requirements for 10.5.1 do say IE9 and upwards, but there's little guidance that says what error messages and failure types you will get with incompatible browsers and why this happens, plus any FAQs to solve these problems. I'd imagine the problem isn't that common but an ESRI knowledge base article on it would be nice.

We tested the fact the IE9 and up browser has to be installed on the host machine by the fact we tried to launch the upgrade through IE11 on a remote machine and the task still failed. However, after installing IE9 on Windows Server 2008 Standard SP2 the upgrade completed successfully.

I might add for anyone using Windows Server 2008 Standard SP2 that upgrading to a supported browser is no easy job either. Firefox and Chrome no longer support the OS so these are no goes as its not worth the risk using a non-vendor supplied legacy version. IE9 is effectively your only choice and this has three pre-requisite patches that need to be installed first before the IE9 upgrade file will run: KB948465 (Service Pack 2 if you've not yet installed this) then KB971512 then KB2117917.

You may also need to use an Internet Explorer 9 offline installer from the Microsoft website since the other Microsoft website downloads found from Google didn't run for us saying that the installer did not support our operating system. Note the IE9 installers are labelled "Windows Vista" but that's fine as Windows Server 2008 SP2 is based on Vista, the newer Server 2008 R2 was based on Windows 7.

Happy to tackle any questions for someone with a similar issue.

Thanks

Ryan

LudwinPoppe3 · ‎02-26-2018

Hi,

I have a similar problem with arcgis server 10.5.1 on windows server 2008R2 SP1:

Unfortunately, upgrading to IE9 didn't solve the problem.

Unlike you I cannot reach manager while doing the upgrade as it fails immediately (even when running Asynchronously).

Any ideas which other options are available?

RyanMartin2 · ‎02-26-2018

Your scenario seems likely to be totally different (based on error message and different OS) and maybe warrants an independent thread of discussion. Nonetheless I'd point you towards the server logs in the first instance.

In your default log location check the logfile for ArcGIS Server. It's user configurable but mine is at C:\<Custom Folder Structure>\logs\<machine name>\server\

This is the installation directory that holds the config files, logs, etc, for the website which is user specifiable during initial setup, it is not necessarily in Program Files though the default location is <ArcGIS Server installation directory>/arcgis/server/usr/config-store which would be in Program Files.

You can probably find the directory (if you don't know where it is) by searching the drive where ArcGIS was installed (unless the config store was put somewhere else, it is possible to do that) for "arcgis-logsettings.json" and then doing "open file location" on that result (the type RLOCK File result), go back one level and you should see config-store, directories and logs. Drill down into the logs folder (as described in 2nd paragraph) and open the most recent file. I found opening with notepad and pasting into word helped with readable formatting.

ChrisSmith7 · ‎02-28-2018

I ran into this same issue while upgrading from 10.3 to 10.6. After opening a case with Esri, they had my apply some permissions changes; however, this did not resolve the issue. After a bit of research on my own and realizing upgrade scripts are running through the browser, I went ahead and made the following changes:

* cleared all cache/browser data in IE

* used the machine name instead of localhost to access arcgis manager, e.g. https://gisserver001:6443/arcgis/manager/

* added the manager site (referenced above) to the trusted sites list in IE internet options

* restarted IE11 as an administrator (c:/ >> program files(x86) >> Internet Explorer >> iexplore.exe)

* continued upgrade process

...and, voila! The upgrade was successful! Hope this helps someone... I'm not sure if it was a combination of actions or a single action that made the upgrade work, but it did!