ArcGIS Web Adaptor 11.1 App Pool freezes

[Scott_Tansley]

Hi.   

I've recently upgraded a client to ArcGIS 11.1, and we're having random problems with the new web adaptors.  I'm looking to see if anyone has observed the same issues.

So the clients ArcGIS Enterprise was deployed at 10.8.1.  There's an IIS Web Server in the DMZ.  A single host with the rest of the base deployment, which is only used for Hosted Feature Services.  WA's exist for portal and hosting.  There is a third machine with a general-purpose ArcGIS Server, federated and primarily serving Map Image Layers.  There is a Web Adaptor called server.

There have never been any repeated outage issues.  The environment was upgraded to 10.9.1 last year.  Once again no issues.

They were upgraded to 11.1 two weeks ago.  Immediately, we found that the machine was running out of memory, we noted the advice given in the new dependencies, and increased the RAM from 4 to 8GB.  It sits at around 5-6GB with no issues, and we have not seen any spike above 6GB to date.

After adding RAM it all seemed to settle for a few days.  But now, every couple of days the IIS application pool for the 'server' web adaptor will just stall.  IIS logs show 200/304 responses for everything up to the freeze/stall and 500 for everything.  There is nothing untoward in the requeste.

ArcGIS Server is still available on 6443 and can be accessed.  It just isn't receiving requests from IIS.  With Info logging turned on, it shows the last good 200 request from the WA.  Then nothing, no errors, no issues.  It's just as if it's sat there waiting for a request and not receiving it.

There have been no firewall or environmental changes recently, the only change is the upgrade to 11.1 and the addition of memory.

On the web server there is nothing in event viewer, system/admin/security or IIS application logs.

I'm blind.  It's just as if the App Pool WA says I've had enough.

The only way to bring the application back online is to restart IIS.  On the AppPool you can stop it.  But it will not start unless IIS is restarted. 

I'm currently blind.  We've external ping monitoring in place so we know when the healthCheck API fails, but there's nothing else we can do but monitor and restart at this point.

[RyanUthoff]

This has been a disaster for us. We went from 10.9.1 where we experienced no issues at all, and upgraded to 11.1 and have been experiencing multiple outages each day. We have installed both web adaptor patches and it has still NOT resolved this issue.

Restarting IIS immediately fixes the issue, so I'm betting IIS and the web adaptor is the issue here and we're still impacted by this bug. Has anyone else installed the second patch and still experiencing issues?

[LukeSavage]

Haven't had an issue since first patch install on 8 different Enterprise environments.  Did you try the workaround?  kick up the queues to 100000 to see if that works if the workaround times out eventually.  We installed the second patch and still no issues on our end with 3 separate ArcGIS Enterprise environments.  Albeit, it's not used as much as the sales production environment as we are still on 10.9.1 in that environment.  Everything else is being tested in 11.1.

[RyanUthoff]

I have not tried the workaround yet because I wanted to give the Esri patches an opportunity to fix the problem on its own. Also, someone from Esri said that the workarounds only delay inevitable outage, not completely eliminate it. 

I'll definitely try the workaround because less outages is certainly better than doing nothing, but honestly, we just need the web adaptors to work correctly and not have to implement workarounds that just delays the inevitable. (I'm sorry, the frustration is not directed towards you. I appreciate your comment!)

[mwartman_grey]

I can also report that we installed both web adaptor patches and the issue has still NOT been resolved 😢. I tried rebooting the server machine after the latest patch install but that didn't help. 

Just curious, are your map services referencing enterprise geodatabases that are a few versions old? I'm wondering if there might be a version incompatibility between the 11.1 web adaptor and enterprise geodatabase data source.  

[RyanUthoff]

Thank you for your reply and validation that I'm not the only one who is still experiencing issues even after installing both patches. We have varying EGDB versions, but most are not the most recent version. Most are around 10.7.1 or 10.8. If there is a compatibility issues with the web adaptor though, I'd like to see that documentation from Esri.

[Edgar_W_Iparraguirre]

IMHO,  there is not a single issue with Web Adaptor 11.1 but many, all being triggered by different causes. Many of our customers do not experience the issue, but many others do. So some issues may be rooted at the Windows environment (be OS, IIS version, security, etc), so not always an issue will be related to the software itself.

To my surprise, on one my new virtual machines I have installed 11.1 from scratch ... and just hit the jackpot, but not on another machine ... all very peculiar/odd/strange and contributing to getting bolder ...

[JonEmch]

Hello there,

  Could you comment on whether or not the issues has changed at all since installing the second patch?

[JasonHarris2]

Hasnt resolved it for me either.  Back to Luke's fix.

[JonEmch]

Hello Jason, as with Ryan above, has the behavior changed at all? Have you engaged with our tech support staff?

[JonEmch]

Hello Ryan, thank you for the feedback on the second patch. I wanted to follow up on this, could you comment on the specific issue you are running into the the Web Adaptor? As in does it change at all between the installation of the first or second patch? I would like our tech support staff to take a look at this issue, please let me know if you have engaged them so that I can get more context there.