nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Upgrade from 11.3 to 11.4 broke shared instances

tcaulum_WPPI

I recently upgraded our environment from 11.3 to 11.4. I keep up with latest releases as best I can, usually within a month or two. It's not a complicated setup so upgrades are straightforward. This time, after what looked like a successful upgrade, most of our services were not accessible.

I attempted the upgrade twice with the same result before realizing only the shared instances were not loading or accessible via REST. I searched for any reason at all why this is happening but found nothing. On a whim, I changed the settings for our shared instances from 8 shared instances per machine to 4. For some reason this fixed it. I even changed it back to 8 and everything is working fine.

Just wanted to make a post in case anyone else found this same issue.

Najděte více příspěvků označených jako

Accepted answers

DenisLalonde

Recently upgraded to 11.4 (from 10.9.1). After a scheduled windows server reboot, we had the same issue with services in the shared instance pool not functioning. With the help of Esri.ca Support, was able to get those services running again by stopping and starting the DynamicMappingHost in Server Admin.

I tried rebooting the server again and was able to recreate the issue, but the stop/start of DynamicMappingHost fixed it again... So, I'm now looking to make sure all services in the pool are working properly and that none of them are bogging everything down on system startup.

All comments

Yogesh_Chavan

Hi @tcaulum_WPPI,

What was this setting before the upgrade? Do you recall if this count was different in 11.3 and upgrading to 11.4 changed this to default or something else?

Also, as a side question, we are on 11.3 and as per the docs ESRI had updated its licensing for ArcGIS Pro, and also updating user types with new names and feature. Does it impact in any ways with the upgrade process? Or its the usual workflow we can follow for an successful upgrade?

tcaulum_WPPI

It was on 8 shared instances per machine while we were on 11.3 and we never had any issues with it. That setting didn't change during the upgrade, still not sure why it choked it up.

The changes in licensing format did not impact the upgrade in any meaningful way. There are prompts in post-processing of Portal saying "hey you have some duplicate licensing now, want us to automatically remove that?" but the wizard was able to convert everything else. If I hear from a user that they're not allowed to access something they used previously, I'll let you know.

DavidColey

Hey @tcaulum_WPPI - can you tell me if you guys were on a concurrent license at 11.3? I'm getting ready to move to11.4, but we are still on the concurrent license manager at 11.3 for our Pro3.4 users (mostly because we don't have the funding yet to purchase the Professional and Professional Plus user types for GIS staff).

So I'm trying to figure out the implications... thanks

tcaulum_WPPI

Hey David, we have fully converted to username licensing. However, there is a new version of License manager for Pro 3.4, leading me to believe you can still use it for concurrent licensing, even in Enterprise 11.4.

DavidColey

Thanks for the reply @tcaulum_WPPI - actually we're good. I forgot about the conversion mechanism where we are able to convert 1 concurrent Creator to up-to 3 named user types, although I have to double-check the levels. Yes you are correct, we can still use the concurrent use license manager at 11.4, it is just not required

tcaulum_WPPI

Excellent! Though moving on from concurrent licensing takes a little bit of getting used to, it's got some benefits when your users get comfortable with it.

AdrianRicketts_GNL

Our organization also observed this behavior when upgrading to ArcGIS Enterprise 11.4 from 11.3. We've had an open ticket with ESRI, and one temporary solution was to gracefully stop the ArcGIS Server and restart the ArcGIS Server. This had worked for our staging environment, but not for production.

Our temporary fix was to configure services to dedicated instances, but this is a short-term solution until the issue is addressed appropriately.

I performed the same fix as yourself - decreased the number shared instances per machine from 8 to 4, and then back to 8. This did the trick - great find! Odd, but great find for others that have others that have observed this issue.

DenisLalonde

TownofWhitbyGISServices

Thank you so much. I also had this issue and stopping and starting the DynamicMappingHost fixed the issue.

DenisLalonde

A bit more info that seems to have solved the issue on my end when rebooting our server:

- Set ArcGIS Datastore, Portal, Server services to manual

- In Scheduled Tasks on system startup, via a .bat file, stagger the ArcGIS component startup leaving a full 10 minutes for Portal to start before starting the ArcGIS Server service.

- In my case, after system starts up, wait 5 minutes, then start Datastore, wait 5 minutes, then start Portal, wait 10 minutes, then start Server

- I have always had a 5-minute stagger between each, but after the upgrade to 11.4, it looks like Portal needs more time than that now

- This issue was logged as a defect and is being reviewed by ESRI product teams.

DavidColey

Yeah, I had some wankiness on this as well, but not until we performed our restarts as part of the Windows Updates and then restarts. We too always restart in a stagger, with our config server going first, then host site server 1, host site server 2, relational data store, object store, fed site server 1, fed site server 2, and then portal server.

So being aware of the Dynamic Mapping host service not behaving, I wasn't sure how that was going to manifest on our system. For the host site, it's not a big deal as we only use the host site for hosted feature layers, vectortile layers, scene layers...

For the Federated site, it does matter. That's the site, (for us) hosting referenced feature access services in shared instances. After our restarts, I could see the dynamic mapping host was running on federated server 1, but was not running on federated server 2.

After I restarted federeated server 2, dynamic mapping host came back up and so was visible as running on both machines. Then I went in to site settings, decreased to 4, saved. Then increased back to 8 (per machine) saved again. Then for server manager for the federated site, I restarted the dynamic mapping host service. It came back up showing 16 available max instances.

So that did it for us, but I will be checking it's status after next months windows updates / server restarts

tcaulum_WPPI

We have since noticed that every time we reboot the server (once a week for maintenance purposes) the same issue crops up. It's an easy fix, but if ESRI comes back with a long term solution let me know. I'm close to opening a ticket with them, as we have to manually reboot the servers now.

tcaulum_WPPI

I'll test this as well, this would be easier to automate than my solution. Let me know if a permanent solution arises. It's starting to feel like something that won't have a fix until 11.4.1 unless its widespread enough to make a separate patch. I'll open up a bug report for it.

David_Brooks

Interestingly, we've seen really poor performance on our shared instances since upgrading from 10.9.1 to 11.3. All the services on shared are taking ~10-20seconds to render, but if we switch them to Dedicated with 0 min instances, they perform beautifully, and we can set the idle time to 30 or 60 seconds to release the resource.

I have tried the suggestions above including cycling the 4/8 shared instance setting, but none of this seems to resolve the issue.

Next stop ESRI.

DenisLalonde

FYI - through my support case with ESRI last month, it was determined that this was indeed a bug with DynamicMappingHost and logged under BUG-000173043. It's currently "in product plan".

Anxiously awaiting a quick fix to this 🤞.

David_Brooks

@DenisLalonde that's interesting! which version did you have the bug logged against? and did they say if it's for multiple versions?

DenisLalonde

@David_Brooks, we upgraded from 10.9.1 to 11.4, so I'm guessing it would have been logged against 11.4. Sorry, no idea RE multiple versions.

MattiasEkstrom

@David_BrooksWe have the same problem after upgrading from 10.9.1 to 11.3, and rebooting, or restarting DynamicMappingHost or altering shared instance settings does't resolve our issue.
Is your ArcGIS Server log also filled with a lot of "Instance of the service 'System/DynamicMappingHost.MapServer' crashed. ..." messages?

David_Brooks

@MattiasEkström glad to hear it's not just us, although Im sorry for you guys too. No our logs are actually very quiet on the whole. This week i wrote a python script that converts our shared and dedicated instances to "dedicated with 0 min and 2 max" and changes the idle time from 1800 to 60 seconds. This seems to have dramatically improved performance. We're currently monitoring peak periods as we expect the system to struggle with so many dedicated instances potentially firing up. So far it's much better.

I have a meeting with ESRI Germany today on this topic, so will report back anything of value.

MattiasEkstrom

We have increased RAM on our server and reduced the number of services so we have been able to change all our shared services to dedicated, which solves or works around the problem. According to tech support our issues is also related to BUG-000173043 and we're at 11.3.

RonnieRichards

We are having the same problem once we upgraded from 11.3 to 11.4 with DynamicMappingHost. Although additionally for us many of the standard system and utility soc's are not starting either. If we give the systems some time and restart ArcGIS Server after about 8-10 minutes everything comes up ok.

In a brand new install of 11.4 this startup symptom did not exist but as soon as we added a registered data store for a SQL Enterprise gdb then this situation started occurring. We delete the registered data store item from portal, restart and everything comes up healthy?

So in these tests we have not even implemented any custom services using SHARED instances yet. We have delayed the start of ArcGIS Server service which helps get the services healthy and not have to baby sit after reboots but we need a permanent fix.

DenisLalonde

Hey guys, I received an update regarding BUG-000173043. It looks like there's a permanent fix coming in version 11.5. Not sure if there will be a patch in the interim - but I'm hoping.

RonnieRichards

Thanks for this @DenisLalonde ! This is very helpful but does not align with exact scenario we are running into.

Curious for your environment are you using any Registered Data Stores (Database)?

RonnieRichards

ESRI support has confirmed the issue we are dealing is related to the same bug. Our automated work-a-round for this was to restart the ArcGIS Server windows service after 10 minutes of windows start-up.

We then deleted all of the portal registered data stores items and this problem went away for us. The backend registered data stores in ArcGIS Server still work for our environment.

KevinPiraino2

Denis, was ESRI technical support ever able to give you an answer about why restarting the DynamicMappingHost service resolved the issue or did they provide you a long term solution if this continues to occur?

DenisLalonde

Other than confirming it as a bug @ 11.4, no specific reason provided. Bug has been resolved at 11.5.

We're still actually @ 11.4. The long-term solution that's still working for us was boosting my DynamicMappingHost pool to 8 and staggering service startup by 10 minutes at server reboot (datastore-portal-server).

Připojte se ke konverzaci

Přihlášení členové mohou přispívat, sledovat aktualizace a další. Jste tu noví? Zaregistrujte si bezplatný účet.

Rychlé odkazy

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.