Upgrade of Hosting GIS Server from 10.7.1 to 10.8.1 Fails During "Continue Server Upgrade"

[GavinCollins1]

We have tried to perform an upgrade of a ArcGIS Enterprise (10.7.1) portal which comprises of a base deployment and an additional federated GIS server. During the upgrade of the Hosting GIS Server, the setup.exe completed successfully however when we attempted to compelte step 6 described in the upgrade documentation, an error is repeatedly returned when we click Continue Server Upgrade.

The error message is "ArcGIS Server upgrade step 2 of 2 : Failed. Refresh auto-deployed services during upgrade." 

In the ArcGIS Server logs we found an anomaly entry for the OfflinePackaging.GPServer being corrupt in the config-store. We repaired this file after noticing it contained malformed text compared to the expected parameters such as the description of the service however the Continue Server Upgrade continues to present the same error message. Historic arcgis server logs indicate this issue existed prior to the upgrade being ran.

ArcGIS Server Logs:

<Msg time="2020-09-03T10:24:38,841" type="INFO" code="7537" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Upgrade step 1 of 2 : Started.</Msg> <Msg time="2020-09-03T10:24:38,849" type="INFO" code="7537" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Upgrade step 1 of 2 : Removing local configurations.</Msg> <Msg time="2020-09-03T10:24:38,877" type="INFO" code="7537" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Upgrade step 1 of 2 : Removed local configurations.</Msg> <Msg time="2020-09-03T10:24:38,879" type="INFO" code="7537" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Upgrade step 1 of 2 : Machine certificates and SSL logic started.</Msg> <Msg time="2020-09-03T10:24:38,885" type="INFO" code="7295" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Validating key store</Msg> <Msg time="2020-09-03T10:24:49,952" type="INFO" code="7537" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Upgrade step 1 of 2 : Machine certificates and SSL logic completed.</Msg> <Msg time="2020-09-03T10:24:49,953" type="INFO" code="7537" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Upgrade step 1 of 2 : Web server starting.</Msg> <Msg time="2020-09-03T10:24:57,424" type="SEVERE" code="6652" source="Admin" process="31964" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Unable to read the service 'OfflinePackaging.GPServer' from the configuration store. A JSONObject text must begin with '{' at character 1 of </Msg> <Msg time="2020-09-03T10:24:57,663" type="INFO" code="7565" source="Server" process="31964" thread="27" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Recycling is configured for service Utilities.Geometry.GeometryServer. Recycling will start at 04/09/2020, 00:00 and repeat every 24 hours.</Msg> <Msg time="2020-09-03T10:24:57,894" type="INFO" code="7565" source="Server" process="31964" thread="27" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Recycling is configured for service Utilities.GeocodingTools.GPServer. Recycling will start at 04/09/2020, 00:00 and repeat every 24 hours.</Msg> <Msg time="2020-09-03T10:24:57,895" type="INFO" code="7565" source="Server" process="31964" thread="27" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Recycling is configured for service System.DynamicMappingHost.MapServer. Recycling will start at 04/09/2020, 00:00 and repeat every 24 hours.</Msg> <Msg time="2020-09-03T10:24:57,913" type="INFO" code="7565" source="Server" process="31964" thread="27" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Recycling is configured for service System.PublishingTools.GPServer. Recycling will start at 04/09/2020, 00:00 and repeat every 24 hours.</Msg> <Msg time="2020-09-03T10:24:57,949" type="INFO" code="7565" source="Server" process="31964" thread="27" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Recycling is configured for service System.PublishingToolsEx.GPServer. Recycling will start at 04/09/2020, 00:00 and repeat every 24 hours.</Msg> <Msg time="2020-09-03T10:24:57,953" type="INFO" code="7565" source="Server" process="31964" thread="27" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Recycling is configured for service Utilities.PrintingTools.GPServer. Recycling will start at 04/09/2020, 00:00 and repeat every 24 hours.</Msg> <Msg time="2020-09-03T10:24:57,961" type="INFO" code="6549" source="Admin" process="31964" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Server machine 'MachineName.Domain' started successfully.</Msg> <Msg time="2020-09-03T10:24:57,976" type="INFO" code="7565" source="Server" process="31964" thread="27" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Recycling is configured for service System.SpatialAnalysisTools.GPServer. Recycling will start at 04/09/2020, 00:00 and repeat every 24 hours.</Msg> <Msg time="2020-09-03T10:24:58,49" type="INFO" code="9011" source="Rest" process="31964" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Application initialized.</Msg> <Msg time="2020-09-03T10:24:58,49" type="INFO" code="9011" source="Rest" process="31964" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Application initialized.</Msg> <Msg time="2020-09-03T10:24:59,295" type="INFO" code="7537" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Upgrade step 1 of 2 : Web server started.</Msg> <Msg time="2020-09-03T10:24:59,295" type="INFO" code="7537" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Upgrade step 1 of 2 : Completed.</Msg> <Msg time="2020-09-03T10:24:59,298" type="INFO" code="7537" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Upgrade step 1 of 2 : Removed the config-store-connection.xml, and created config-store-connection-upgrade.xml for upgrade.</Msg> <Msg time="2020-09-03T10:25:00,422" type="INFO" code="7720" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">The cloud regions configuration file was deleted from this server machine. If you have other server machines in the site, please make sure that the file has been deleted from them as well.</Msg> <Msg time="2020-09-03T10:25:00,588" type="INFO" code="7702" source="Server" process="42044" thread="1" methodName="" machine="MachineName.Domain" user="" elapsed="" requestID="">Web server started successfully.</Msg>

 
Afterwards we researched the error message and found various fixes on the GeoNet which we have attempted with no success;

• Technical Article https://support.esri.com/en/technical-article/000018642 however this applies an upgrade to 10.5.1 therefore non-applicable to us.
• Re-ran 10.8.1 setep.exe as administrator and in repair mode.
• Clearing browser cache.
• Verify ArcGIS Server domain account permissions over its installation and content directory.
• Use a different web browsers - IE11, Google Chrome.
• Run web browser as an administrator.
• Add the fully qualifed domain name of the machine to the trusted sites list in IE.
• Restart the ArcGIS Server service.
• Reboot the machine hosting ArcGIS Server.
• Open Server Manager using the FQDN.

Keen to hear from anybody else who has encountered this error message during the Contine Server Upgrade.

[DanielMurillo]

Hello @GavinCollins1 

I am right now encountering the same issue, were you able to find a solution for that failure?

[DanielMurillo]

A heads up just in case anybody sees this having the issue on version 10.8.1. We resolved the issue by running all patches available on ArcGIS Patch Notification.

Then proceeded to click retry and the process continued and succesfully ended.

[VCGIAdmin]

Seems as if this is fairly common.  I just ran into it last week and I've been going back and forth with Tech Support.  From what I can see in the various threads on this topic, Tech Support it just telling folks to toss IT darts against the install-wall until it works.  What we really need is a separate verbose log for "step 2 of 2" that actually tells us why it failed.  "Step 2 of 2 failed because....STATING EXACTLY WHAT FAILED" Come on Esri, you've got plenty of $ to build a verbose log into this process so we all aren't guessing and tossing IT darts.  Better yet, eliminate the browser insanity! I've never seen a browser used to install server-side software before...definitely a WTF?!