topic Re: Upgrade from 11.3 to 11.4 broke shared instances in ArcGIS Enterprise Questions

Upgrade from 11.3 to 11.4 broke shared instances

tcaulum_WPPI — Fri, 29 Nov 2024 15:54:26 GMT

I recently upgraded our environment from 11.3 to 11.4. I keep up with latest releases as best I can, usually within a month or two. It's not a complicated setup so upgrades are straightforward. This time, after what looked like a successful upgrade, most of our services were not accessible.

I attempted the upgrade twice with the same result before realizing only the shared instances were not loading or accessible via REST. I searched for any reason at all why this is happening but found nothing. On a whim, I changed the settings for our shared instances from 8 shared instances per machine to 4. For some reason this fixed it. I even changed it back to 8 and everything is working fine.

Just wanted to make a post in case anyone else found this same issue.

Re: Upgrade from 11.3 to 11.4 broke shared instances

Yogesh_Chavan — Sun, 01 Dec 2024 06:22:47 GMT

Hi @tcaulum_WPPI,

What was this setting before the upgrade? Do you recall if this count was different in 11.3 and upgrading to 11.4 changed this to default or something else?

Also, as a side question, we are on 11.3 and as per the docs ESRI had updated its licensing for ArcGIS Pro, and also updating user types with new names and feature. Does it impact in any ways with the upgrade process? Or its the usual workflow we can follow for an successful upgrade?

Re: Upgrade from 11.3 to 11.4 broke shared instances

tcaulum_WPPI — Mon, 02 Dec 2024 20:13:23 GMT

It was on 8 shared instances per machine while we were on 11.3 and we never had any issues with it. That setting didn't change during the upgrade, still not sure why it choked it up.

The changes in licensing format did not impact the upgrade in any meaningful way. There are prompts in post-processing of Portal saying "hey you have some duplicate licensing now, want us to automatically remove that?" but the wizard was able to convert everything else. If I hear from a user that they're not allowed to access something they used previously, I'll let you know.

Re: Upgrade from 11.3 to 11.4 broke shared instances

DavidColey — Tue, 03 Dec 2024 14:16:58 GMT

Hey @tcaulum_WPPI - can you tell me if you guys were on a concurrent license at 11.3? I'm getting ready to move to11.4, but we are still on the concurrent license manager at 11.3 for our Pro3.4 users (mostly because we don't have the funding yet to purchase the Professional and Professional Plus user types for GIS staff).

So I'm trying to figure out the implications... thanks

Re: Upgrade from 11.3 to 11.4 broke shared instances

tcaulum_WPPI — Mon, 09 Dec 2024 15:04:02 GMT

Hey David, we have fully converted to username licensing. However, there is a new version of License manager for Pro 3.4, leading me to believe you can still use it for concurrent licensing, even in Enterprise 11.4.

Re: Upgrade from 11.3 to 11.4 broke shared instances

DavidColey — Mon, 09 Dec 2024 17:42:39 GMT

Thanks for the reply @tcaulum_WPPI - actually we're good. I forgot about the conversion mechanism where we are able to convert 1 concurrent Creator to up-to 3 named user types, although I have to double-check the levels. Yes you are correct, we can still use the concurrent use license manager at 11.4, it is just not required

Re: Upgrade from 11.3 to 11.4 broke shared instances

tcaulum_WPPI — Wed, 11 Dec 2024 19:58:55 GMT

Excellent! Though moving on from concurrent licensing takes a little bit of getting used to, it's got some benefits when your users get comfortable with it.

Re: Upgrade from 11.3 to 11.4 broke shared instances

AdrianRicketts_GNL — Thu, 12 Dec 2024 18:19:02 GMT

Our organization also observed this behavior when upgrading to ArcGIS Enterprise 11.4 from 11.3. We've had an open ticket with ESRI, and one temporary solution was to gracefully stop the ArcGIS Server and restart the ArcGIS Server. This had worked for our staging environment, but not for production.

Our temporary fix was to configure services to dedicated instances, but this is a short-term solution until the issue is addressed appropriately.

I performed the same fix as yourself - decreased the number shared instances per machine from 8 to 4, and then back to 8. This did the trick - great find! Odd, but great find for others that have others that have observed this issue.

Re: Upgrade from 11.3 to 11.4 broke shared instances

DenisLalonde — Fri, 13 Dec 2024 14:54:13 GMT

Recently upgraded to 11.4 (from 10.9.1). After a scheduled windows server reboot, we had the same issue with services in the shared instance pool not functioning. With the help of Esri.ca Support, was able to get those services running again by stopping and starting the DynamicMappingHost in Server Admin.

I tried rebooting the server again and was able to recreate the issue, but the stop/start of DynamicMappingHost fixed it again... So, I'm now looking to make sure all services in the pool are working properly and that none of them are bogging everything down on system startup.

Re: Upgrade from 11.3 to 11.4 broke shared instances

JoySoulliere — Tue, 17 Dec 2024 19:24:38 GMT

Thank you so much. I also had this issue and stopping and starting the DynamicMappingHost fixed the issue.

Re: Upgrade from 11.3 to 11.4 broke shared instances

DenisLalonde — Thu, 19 Dec 2024 14:16:29 GMT

A bit more info that seems to have solved the issue on my end when rebooting our server:

- Set ArcGIS Datastore, Portal, Server services to manual

- In Scheduled Tasks on system startup, via a .bat file, stagger the ArcGIS component startup leaving a full 10 minutes for Portal to start before starting the ArcGIS Server service.

- In my case, after system starts up, wait 5 minutes, then start Datastore, wait 5 minutes, then start Portal, wait 10 minutes, then start Server

- I have always had a 5-minute stagger between each, but after the upgrade to 11.4, it looks like Portal needs more time than that now

- This issue was logged as a defect and is being reviewed by ESRI product teams.

Re: Upgrade from 11.3 to 11.4 broke shared instances

DavidColey — Fri, 20 Dec 2024 20:01:11 GMT

Yeah, I had some wankiness on this as well, but not until we performed our restarts as part of the Windows Updates and then restarts. We too always restart in a stagger, with our config server going first, then host site server 1, host site server 2, relational data store, object store, fed site server 1, fed site server 2, and then portal server.

So being aware of the Dynamic Mapping host service not behaving, I wasn't sure how that was going to manifest on our system. For the host site, it's not a big deal as we only use the host site for hosted feature layers, vectortile layers, scene layers...

For the Federated site, it does matter. That's the site, (for us) hosting referenced feature access services in shared instances. After our restarts, I could see the dynamic mapping host was running on federated server 1, but was not running on federated server 2.

After I restarted federeated server 2, dynamic mapping host came back up and so was visible as running on both machines. Then I went in to site settings, decreased to 4, saved. Then increased back to 8 (per machine) saved again. Then for server manager for the federated site, I restarted the dynamic mapping host service. It came back up showing 16 available max instances.

So that did it for us, but I will be checking it's status after next months windows updates / server restarts

Re: Upgrade from 11.3 to 11.4 broke shared instances

tcaulum_WPPI — Thu, 26 Dec 2024 20:48:20 GMT

We have since noticed that every time we reboot the server (once a week for maintenance purposes) the same issue crops up. It's an easy fix, but if ESRI comes back with a long term solution let me know. I'm close to opening a ticket with them, as we have to manually reboot the servers now.

Re: Upgrade from 11.3 to 11.4 broke shared instances

tcaulum_WPPI — Thu, 26 Dec 2024 20:50:47 GMT

I'll test this as well, this would be easier to automate than my solution. Let me know if a permanent solution arises. It's starting to feel like something that won't have a fix until 11.4.1 unless its widespread enough to make a separate patch. I'll open up a bug report for it.

Re: Upgrade from 11.3 to 11.4 broke shared instances

David_Brooks — Tue, 28 Jan 2025 11:04:39 GMT

Interestingly, we've seen really poor performance on our shared instances since upgrading from 10.9.1 to 11.3. All the services on shared are taking ~10-20seconds to render, but if we switch them to Dedicated with 0 min instances, they perform beautifully, and we can set the idle time to 30 or 60 seconds to release the resource.

I have tried the suggestions above including cycling the 4/8 shared instance setting, but none of this seems to resolve the issue.

Next stop ESRI.

Re: Upgrade from 11.3 to 11.4 broke shared instances

DenisLalonde — Tue, 28 Jan 2025 13:46:00 GMT

FYI - through my support case with ESRI last month, it was determined that this was indeed a bug with DynamicMappingHost and logged under BUG-000173043. It's currently "in product plan".

Anxiously awaiting a quick fix to this 🤞.

Re: Upgrade from 11.3 to 11.4 broke shared instances

David_Brooks — Wed, 29 Jan 2025 11:35:27 GMT

@DenisLalonde that's interesting! which version did you have the bug logged against? and did they say if it's for multiple versions?

Re: Upgrade from 11.3 to 11.4 broke shared instances

DenisLalonde — Wed, 29 Jan 2025 13:56:33 GMT

@David_Brooks, we upgraded from 10.9.1 to 11.4, so I'm guessing it would have been logged against 11.4. Sorry, no idea RE multiple versions.

Re: Upgrade from 11.3 to 11.4 broke shared instances

MattiasEkström — Wed, 29 Jan 2025 14:41:15 GMT

@David_BrooksWe have the same problem after upgrading from 10.9.1 to 11.3, and rebooting, or restarting DynamicMappingHost or altering shared instance settings does't resolve our issue.
Is your ArcGIS Server log also filled with a lot of "Instance of the service 'System/DynamicMappingHost.MapServer' crashed. ..." messages?

Re: Upgrade from 11.3 to 11.4 broke shared instances

David_Brooks — Thu, 30 Jan 2025 11:33:30 GMT

@MattiasEkström glad to hear it's not just us, although Im sorry for you guys too. No our logs are actually very quiet on the whole. This week i wrote a python script that converts our shared and dedicated instances to "dedicated with 0 min and 2 max" and changes the idle time from 1800 to 60 seconds. This seems to have dramatically improved performance. We're currently monitoring peak periods as we expect the system to struggle with so many dedicated instances potentially firing up. So far it's much better.

I have a meeting with ESRI Germany today on this topic, so will report back anything of value.