ArcGIS Web Adaptor 11.1 App Pool freezes

[Scott_Tansley]

Hi.   

I've recently upgraded a client to ArcGIS 11.1, and we're having random problems with the new web adaptors.  I'm looking to see if anyone has observed the same issues.

So the clients ArcGIS Enterprise was deployed at 10.8.1.  There's an IIS Web Server in the DMZ.  A single host with the rest of the base deployment, which is only used for Hosted Feature Services.  WA's exist for portal and hosting.  There is a third machine with a general-purpose ArcGIS Server, federated and primarily serving Map Image Layers.  There is a Web Adaptor called server.

There have never been any repeated outage issues.  The environment was upgraded to 10.9.1 last year.  Once again no issues.

They were upgraded to 11.1 two weeks ago.  Immediately, we found that the machine was running out of memory, we noted the advice given in the new dependencies, and increased the RAM from 4 to 8GB.  It sits at around 5-6GB with no issues, and we have not seen any spike above 6GB to date.

After adding RAM it all seemed to settle for a few days.  But now, every couple of days the IIS application pool for the 'server' web adaptor will just stall.  IIS logs show 200/304 responses for everything up to the freeze/stall and 500 for everything.  There is nothing untoward in the requeste.

ArcGIS Server is still available on 6443 and can be accessed.  It just isn't receiving requests from IIS.  With Info logging turned on, it shows the last good 200 request from the WA.  Then nothing, no errors, no issues.  It's just as if it's sat there waiting for a request and not receiving it.

There have been no firewall or environmental changes recently, the only change is the upgrade to 11.1 and the addition of memory.

On the web server there is nothing in event viewer, system/admin/security or IIS application logs.

I'm blind.  It's just as if the App Pool WA says I've had enough.

The only way to bring the application back online is to restart IIS.  On the AppPool you can stop it.  But it will not start unless IIS is restarted. 

I'm currently blind.  We've external ping monitoring in place so we know when the healthCheck API fails, but there's nothing else we can do but monitor and restart at this point.

[LukeSavage]

Awesome @JasonHarris2 .  Hope it holds.  esri is now testing the configuration on their end. Esri Case #03353671 - ArcGIS Web Adaptor 11.1 App Pool freezes

[MichaelSnook]

Hi All

I have been fighting the exact issue that @Scott_Tansley outlined for about 3 weeks now (with support) and no solution.  Everything has been working fine for the last 8+ years until we upgraded from 10.9.1 to 11.1.  Just random crashes/freezes that only seems to affect Server...Portal still functions and the overall webserver is healthy.  Incredibly frustrating that it just stops responding -- with no obvious reason.

I just found this thread last night and I'm testing out the IIS config suggestions from @LukeSavage . 

Fingers crossed!

I'll report back.  Thanks!

[MichaelSnook]

So far so good...made it through the first day since May 6 (excepting weekends) without a crash!  

[Anonymous User]

Has this issue been fixed?. Is the workaround done by @JasonHarris2 the permanent solution?. Having the upgrade on hold waiting for the resolutions of this issue and the sharing content with OnlyView groups?. Thanks.

[Scott_Tansley]

I can say that my client and I have:

• Raised this as a support call with our local distributor.  I know that it has been escalated to Esri directly and we've recently spoken to Esri support via a Zoom meeting.  We're awaiting official word.
• We've implemented the work around as well, queue lenght and max instances.  The system has been stable/reliable since then.  

Awaiting a hotfix or similar, I'll report as soon as I have anything official.  

[Anonymous User]

Thanks @Scott_Tansley 

[JesseCloutier]

For record keeping: 6/21/2023 post by Anonymous User belongs to @AndresEcheverri 

[MichaelSnook]

I just got notice that:

 BUG-000158629 ArcGIS Web Adaptor (IIS) 11.1 may stop responding to requests while under load.

Has been updated to 'In Product Plan' -- so that's good.

Thanks to all for the help and workarounds!

[JesseCloutier]

For record keeping: 6/21/2023 post by Anonymous User belongs to @AndresEcheverri 

[mwartman_grey]

I encountered this same issue and discovered that one of our heavily used map services with complex polygons (many vertices), covering a large county-wide area was referencing a feature class that had a bad/incorrect spatial index in the SQL Server geodatabase.  After rebuilding the spatial index, recalculating the spatial extent and validating the index on this particular feature class, the web adaptor issues stopped occurring!

One thing you could try while we wait for the ESRI fix is to validate your spatial indexes on all referenced feature classes and rebuild them where needed.

There may be other causes to the web adaptor issue, but in my case, my suspicion is that the bad spatial index was 'looking' in the wrong area, and when the requests came in from the web adaptor, the spatial indexing didn’t efficiently return the image. This then caused a request backlog and eventual error. Since indexing is meant to speed up data retrieval, when the index is incorrect it could do the opposite and slow down retrieval. I think the issue was exacerbated by the increased resource requirements for 11.1 web adaptor. Hope that helps.