ArcGIS Web Adaptor 11.1 App Pool freezes

[Scott_Tansley]

Hi.   

I've recently upgraded a client to ArcGIS 11.1, and we're having random problems with the new web adaptors.  I'm looking to see if anyone has observed the same issues.

So the clients ArcGIS Enterprise was deployed at 10.8.1.  There's an IIS Web Server in the DMZ.  A single host with the rest of the base deployment, which is only used for Hosted Feature Services.  WA's exist for portal and hosting.  There is a third machine with a general-purpose ArcGIS Server, federated and primarily serving Map Image Layers.  There is a Web Adaptor called server.

There have never been any repeated outage issues.  The environment was upgraded to 10.9.1 last year.  Once again no issues.

They were upgraded to 11.1 two weeks ago.  Immediately, we found that the machine was running out of memory, we noted the advice given in the new dependencies, and increased the RAM from 4 to 8GB.  It sits at around 5-6GB with no issues, and we have not seen any spike above 6GB to date.

After adding RAM it all seemed to settle for a few days.  But now, every couple of days the IIS application pool for the 'server' web adaptor will just stall.  IIS logs show 200/304 responses for everything up to the freeze/stall and 500 for everything.  There is nothing untoward in the requeste.

ArcGIS Server is still available on 6443 and can be accessed.  It just isn't receiving requests from IIS.  With Info logging turned on, it shows the last good 200 request from the WA.  Then nothing, no errors, no issues.  It's just as if it's sat there waiting for a request and not receiving it.

There have been no firewall or environmental changes recently, the only change is the upgrade to 11.1 and the addition of memory.

On the web server there is nothing in event viewer, system/admin/security or IIS application logs.

I'm blind.  It's just as if the App Pool WA says I've had enough.

The only way to bring the application back online is to restart IIS.  On the AppPool you can stop it.  But it will not start unless IIS is restarted. 

I'm currently blind.  We've external ping monitoring in place so we know when the healthCheck API fails, but there's nothing else we can do but monitor and restart at this point.

[Scott_Tansley]

This is interesting.  We also saw content related isssues.  A print service was called that referenced a tile map cache for urban imagery.  The print was outside of the rural area.  The cache was rebuilt to include rural imagery with urban overlaid.  The number of issues reduced after the rebuild.

Also many Map Image Layers were built from file geodatabses.  Feature access was turned on after publishing. But feature access requires an enterprise geodatabase.  This also appeared to cause problems when the service was consumed.

remedying these two issues reduced a lot of the outages.  However they continued until we changed the max instance and max queue length values as discussed in this topic.  Those changes finally brought stability.  

We had reported the ‘content observations’ to Esri support but thought it was a red herring, so this is valuable information.

[DavidColey]

I just want to report that this still has not been an issue for us.  I'd say I was one of the lucky ones, but I'd rather say that our distributed deployment on Windows Server 2016 and all network protocols (open ports, and firewall rules, group policy rules for accounts, IIS roles and features, server roles and features, etc) are responsible for our continued stability. 

I have not had to adjust max instance and max queue length values.  Currently the web server is running 4 w3wp processes.

We use f5 as our reverse proxy. The f5 does no load balancing at all, it just directs requests. The web (4-core 16Gb Ram) server does not use a DNS alias, rather it is aliased on the f5. We do this so that all machines that do participate in our deployment - portal, hosting site, federated site, 2 data stores - can use our CA wildcard certificate and validate as secure when the certificate is applied to each machines' DNS.  

Our configuration server (config store, server directories) has no issues communicating with all machines that participate in the site.

UPDATE: Still no issues for our site after applying the second patch. on 08/15/23

Thanks,

David

David Coley

ArcEnterprise Administrator

Sarasota County GIS

[SimonSchütte_ct]

Looks like a patch is out: ArcGIS Web Adaptor (IIS) 11.1 Reliability Patch (esri.com)

[Scott_Tansley]

Nice discovery.  Excellent news all around.

[MichaelSnook]

Unless I'm completely missing it, there doesn't seem to be a link to download the patch anywhere on that patch info page.  Does anyone see it anywhere?

[Scott_Tansley]

the primary link to the patch description now resolves to a 404

[AndrewBowne]

Add me to the list of people experiencing the same issue.  

I also do not see a link to the patch anywhere.  I tried patchfinder.exe on the server and nothing shows up for the web adaptor.  Anyone else able to find it?

[SimonSchütte_ct]

Checksum      (SHA256)

XXXX

gisupdates.esri.com/QFE/WAI-111-P-942/ArcGIS-111-WAI-R-Patch.msp

Probably prematurely published. It should be available soon.

[TonyAlfredsson_]

Multiple installations with this issue, different combinations like everything on the same server, and DMZ with wepadaptor installations. Hope they re-release the patch soon.

[asergio]

We are having the sames issues. Waiting for the ArcGIS Web Adaptor (IIS) 11.1 patch.