Error Upgrading ArcGIS Server from 10.9.1 to 11: ArcGIS Server upgrade step 2 of 2: Failed. Refresh auto-deployed services during upgrade

Cory_Ott · ‎08-17-2022

Hi Everyone! We have been attempting to perform an upgrade on our ArcGIS Portal & Server from version 10.9.1 to 11 over the past couple of weeks, and consistently encounter the following error message when attempting to finalize Configuring ArcServer.

ArcGIS Server upgrade step 2 of 2: Failed. Refresh auto-deployed services during upgrade

This occurs in both our staging and production environments. We have been working with ESRI Tech Support for the past 2 weeks to try and resolve what is causing this error to be thrown at the final stages of refreshing the auto-deployed services during the upgrade. There have been approximately 5-6 different solutions attempted by ESRI to resolve this error, which have been heavily focused on permissions and IIS configurations, but there has not been any success to date. We have been able to replicate the error across both environments with and without ESRI tech support's assistance. When researching this error, it has occurred in the past when upgrading from multiple different versions, but none of the solutions that have worked for past upgrades have not worked for the upgrade to version 11.

Has anyone else encountered this issue while trying to upgrade from 10.9.1 to 11 this year? Any advice, comments, thoughts, and possible solutions would be greatly appreciated! Thank you all for your time!

ArcGIS Server upgrade step 2 of 2: Failed. Refresh auto-deployed services during upgrade

KurtisGagne1 · ‎08-17-2022

I'm experiencing this identical situation! Upgrading 10.9.1 to 11. I've tried a few things so far but nothing has been working. I'll see if I can post more information when I have more time.

Do you have a case number or bug number with esri?

UPDATE:

Had a chance to dig deeper.

So it looks like my ARCSOC.exe's are crashing immediately after click the 'Upgrade' button. When I go to Windows Event Viewer it identifies a DLL (gdal_e.dll) in the ArcGIS server directory as the culprit.

Out of curiosity could you have a look at your windows Even Viewer the moment after the Upgrade fails to see if there's any errors mentioning 'gdal_e.dll'? If you have a similar error somewhere in there, we may truly be experiencing the same bug. You may have to sift through the Event Viewer, tends to have a lot of entries.

Cory_Ott · ‎08-22-2022

Hello Kurtis,

Thank you very much for reaching out, I appreciate you sharing your experience with me! I do have a case number with ESRI that has been open for about 3 weeks on this issue. I may be able to provide some more information about our situation later today when I can find some time.

That's very interesting to learn about the ARCSOC.exe's crashing immediately when trying to push the server upgrade. I do not believe that ESRI tech support has attempted this potential solution yet, so I will share this information with the specialist that I have been working with so we can troubleshoot this option. Next time that I attempt this upgrade on our staging server (hopefully early this week), I will be sure to capture screenshots of the Windows Event Viewer immediately after trying to finalize the server upgrade configuration.

Thanks again for bringing this information to my attention, anything and everything helps narrow down the error and issue! I will reach back out on this thread with any additional information or experiences that I encounter after the next troubleshooting session with ESRI. I'm somewhat glad that I am not the only one experiencing this issue while trying to upgrade to 11.0. Thanks again!

Best regards,

-Cory Ott

JonEmch · ‎08-18-2022

Hello Cory,

I am sorry to hear that you are having issues upgrading ArcGIS Enterprise. Could you please DM me your case number so that I can review what has been tried and to potentially level some additional resources onto the case?

Keep on keeping on!

Cory_Ott · ‎08-22-2022

Hello Jon,

Sure thing, I will send the case number over to your inbox this morning. We would appreciate any additional information or potential solutions to tackling this error. Thanks for your time!

-Cory Ott

DavidColey · ‎08-29-2022

Hello - Is this happening on trying to upgrade the hosting site or a federated site (if distributed)? Did you make sure that all of your Content and Services directories are enabled and all the site admin accounts are enabled?

I did experience an issue where server did not complete an upgrade at an older 10.x to 10.x.something because the web adaptor did not completely unregister from server after it was un-installed from our web server machine. I do not recall the specific step it failed on, but it was probably past this step 2. But once the web adaptor was completely unregistered from Server Admin, the upgrade completed.

At the 10.9 to 10.9.1 upgrade, I did have to modify the pg_hba.config file:

# TYPE DATABASE USER ADDRESS METHOD

host all all ::1/128 scram-sha-256

to handle the additional encryption method.

Lastly, did you try not using localhost in the manager url but/and/or machinename:6443 or DNSname:6443 ?

To me, Soc's crashing sounds like a windows arcserver service account permissions thing . . .

@Cory_Ott @KurtisGagne1 @JonEmch

KurtisGagne1 · ‎08-29-2022

This may not match what others are experiencing, but thought I'd post a reply just in case.

After a lot of sleuthing it's looking like I'll need to take this up with support, here's what I found (and a very bizarre workaround):

During the post-installation upgrade step, the web interface threw an error:

I found several posts suggesting it was permissions or related to the service user that ArcGIS Server runs as.

None of those solutions panned out.

Then I noticed that when I clicked Continue Server Upgrade the task manager would populate with ARCSOC.exe services slowly, then nearly all at once they would disappear and the web interface would throw the error.

Searched to ArcGIS server logs, but there was nothing to indicate the services had even started.

Then l had a look through Windows Event Viewer and found tons of application errors generated by ARCSOC.exe, with the faulting module gdal_e.dll.

Had a look at the DLL file itself and it turned out to be a very newly compile version of GDAL.

I decided to compare it to the DLL included with ArcGIS Pro 2.9.

Noting that it wasn't a major version change, I decided to risk it and replace the 3.4.0.34 version with the 3.3.0.25 from Pro 2.9.

After doing that, the upgrade completed successfully!

It seems that this particular build of the GDAL DLL 3.4.0.34 might have an issue running on our VM.

This wouldn't be the first time we've found a broken build of a DLL. We had issues with the version of NumPy ESRI includes with the python environment, where it would crash specifically on the particular CPU used by our VM because it had been built assuming support for a particular CPU feature.

Mixing DLLs versions like this is of course not ideal, especially on a production machine.

DavidColey · ‎08-30-2022

Thanks for the feedback @KurtisGagne1 . Yes, a bizarre workaround indeed. I have never encountered anything like this since we first stood up Enterprise at 10.6 and agree that this is very risky.

Sorry - I have a few questions if you don't mind - why or how would you ever think to compare a gdal dll that is part of the Server framework with one that is contained in Pro - do you have Pro installed on the same machine as Server?

Is this a base or distributed deployment, and if distributed, did this happen on your hosting site or federated site?

Update:

The current gdal file in our current Server 10.9.1 is named gdal203e.dll version 2.3.3.38 - so not the same file, at least as far as naming goes.

@JonEmch - do you have anything to add?

DavidColey · ‎08-30-2022

Update:

Some more digging. As I said, the current gdal file in our current Server 10.9.1 is named gdal203e.dll version 2.3.3.38 and exists as part of the ArcMap-based runtime.

This file exists on our federated site where we did not disable the ArcMap-based runtime when we upgraded from 10.9.0 to 10.9.1 to support 2 map services containing the Geometric Network.

We did disable the ArcMap-based runtime on our hosting site when we upgraded from 10.9.0 to 10.9.1 and so none of the ArcMap-based runtime files exist in the bin folder of the hosting site.

While probably not possible for everyone, when we are able to discontinue using the Geometric Network, we had planned on just deleting those services from the federated site before upgrading that site to 11, as everything else on the site was authored in Pro.

In the meantime, we are leaving the federated site at 10.9.1 until our UN is ready to go, and even then the federated site will probably have to remain 10.9.1 until we can move to UN version 6

Also - we did have install the .Net 6 desktop-runtime framework on all 4 vm servers running ArcServer in both our hosting and federated sites.

Hopefully this helps-

@JonEmch

JonEmch · ‎08-30-2022

Thank you for pining me on this, still exploring this on my end, but the license file difference is an interesting angle. Will report back if I find anything to add.

Keep on keeping on!