Upgrade from 11.3 to 11.4 broke shared instances

[tcaulum_WPPI]

I recently upgraded our environment from 11.3 to 11.4.  I keep up with latest releases as best I can, usually within a month or two. It's not a complicated setup so upgrades are straightforward.  This time, after what looked like a successful upgrade, most of our services were not accessible. 

I attempted the upgrade twice with the same result before realizing only the shared instances were not loading or accessible via REST.  I searched for any reason at all why this is happening but found nothing.  On a whim, I changed the settings for our shared instances from 8 shared instances per machine to 4.  For some reason this fixed it.  I even changed it back to 8 and everything is working fine.  

Just wanted to make a post in case anyone else found this same issue. 

[MattiasEkström]

@David_BrooksWe have the same problem after upgrading from 10.9.1 to 11.3, and rebooting, or restarting DynamicMappingHost or altering shared instance settings does't resolve our issue.
Is your ArcGIS Server log also filled with a lot of "Instance of the service 'System/DynamicMappingHost.MapServer' crashed. ..." messages?

[David_Brooks]

@MattiasEkström glad to hear it's not just us, although Im sorry for you guys too. No our logs are actually very quiet on the whole. This week i wrote a python script that converts our shared and dedicated instances to "dedicated with 0 min and 2 max" and changes the idle time from 1800 to 60 seconds. This seems to have dramatically improved performance. We're currently monitoring peak periods as we expect the system to struggle with so many dedicated instances potentially firing up. So far it's much better.

I have a meeting with ESRI Germany today on this topic, so will report back anything of value.

[MattiasEkström]

We have increased RAM on our server and reduced the number of services so we have been able to change all our shared services to dedicated, which solves or works around the problem. According to tech support our issues is also related to BUG-000173043 and we're at 11.3.

[RonnieRichards]

We are having the same problem once we upgraded from 11.3 to 11.4 with DynamicMappingHost. Although additionally for us many of the standard system and utility soc's are not starting either. If we give the systems some time and restart ArcGIS Server after about 8-10 minutes everything comes up ok. 

In a brand new install of 11.4 this startup symptom did not exist but as soon as we added a registered data store for a SQL Enterprise gdb then this situation started occurring. We delete the registered data store item from portal, restart and everything comes up healthy?

So in these tests we have not even implemented any custom services using SHARED instances yet. We have delayed the start of ArcGIS Server service which helps get the services healthy and not have to baby sit after reboots but we need a permanent fix.

[DenisLalonde]

Hey guys, I received an update regarding BUG-000173043. It looks like there's a permanent fix coming in version 11.5. Not sure if there will be a patch in the interim - but I'm hoping.

[RonnieRichards]

Thanks for this @DenisLalonde ! This is very helpful but does not align with exact scenario we are running into.

Curious for your environment are you using any Registered Data Stores (Database)?

[RonnieRichards]

ESRI support has confirmed the issue we are dealing is related to the same bug. Our automated work-a-round for this was to restart the ArcGIS Server windows service after 10 minutes of windows start-up. 

We then deleted all of the portal registered data stores items and this problem went away for us. The backend registered data stores in ArcGIS Server still work for our environment.