ArcGIS Web Adaptor 11.1 App Pool freezes

[Scott_Tansley]

Hi.   

I've recently upgraded a client to ArcGIS 11.1, and we're having random problems with the new web adaptors.  I'm looking to see if anyone has observed the same issues.

So the clients ArcGIS Enterprise was deployed at 10.8.1.  There's an IIS Web Server in the DMZ.  A single host with the rest of the base deployment, which is only used for Hosted Feature Services.  WA's exist for portal and hosting.  There is a third machine with a general-purpose ArcGIS Server, federated and primarily serving Map Image Layers.  There is a Web Adaptor called server.

There have never been any repeated outage issues.  The environment was upgraded to 10.9.1 last year.  Once again no issues.

They were upgraded to 11.1 two weeks ago.  Immediately, we found that the machine was running out of memory, we noted the advice given in the new dependencies, and increased the RAM from 4 to 8GB.  It sits at around 5-6GB with no issues, and we have not seen any spike above 6GB to date.

After adding RAM it all seemed to settle for a few days.  But now, every couple of days the IIS application pool for the 'server' web adaptor will just stall.  IIS logs show 200/304 responses for everything up to the freeze/stall and 500 for everything.  There is nothing untoward in the requeste.

ArcGIS Server is still available on 6443 and can be accessed.  It just isn't receiving requests from IIS.  With Info logging turned on, it shows the last good 200 request from the WA.  Then nothing, no errors, no issues.  It's just as if it's sat there waiting for a request and not receiving it.

There have been no firewall or environmental changes recently, the only change is the upgrade to 11.1 and the addition of memory.

On the web server there is nothing in event viewer, system/admin/security or IIS application logs.

I'm blind.  It's just as if the App Pool WA says I've had enough.

The only way to bring the application back online is to restart IIS.  On the AppPool you can stop it.  But it will not start unless IIS is restarted. 

I'm currently blind.  We've external ping monitoring in place so we know when the healthCheck API fails, but there's nothing else we can do but monitor and restart at this point.

[DorothyMaguli]

I as well have the same issues with web adaptors at 11.1, the outgates are
so severe.Everytime i do an installation and re installation as well. Any
help accorded to this would of significance

[Scott_Tansley]

Thanks for confirming I'm not alone.  I have asked my client to raise this as an official support ticket.  We have also tried an uninstall/install and that made no difference.  Only restarting IIS is able to temporarily solve this.

[DavidColey]

I have just completed my upgrade from 11.0  / 10.9.1 to 11.1 and can tell you that through Monitor I can now see we have increased the w3wp process count on the web server from 2 to 5.  

We have a distributed deployment on Windows Sever 2016 VMs all 8-core, 32Gb, with the exception of the web server which is at 8-core 16Gb. The web server currently hosts 3 web adaptors for Portal, Hosting Server Site(2-machine), and a Federated Server Site(2-machine). 

I am assuming at 11.1 the additional components for web adaptor - the hosting bundle and web deploy 3.6 are spinning up w3wp processes but I don't know.  I completed the upgrade Saturday 5/21  9 pm EDT, so today is the first weekday under load.

[Scott_Tansley]

Hey David yes.  All web adaptors (prior 10 11.1) ran in a single app pool.  As per Microsoft best practice, each web adaptor is now security isolated in it's own app pool, which increases the RAM required.  I was aware of this.  In our situation it's just one App Pool that freezes, which is the traditional Map Service Server.  It seems to be when it's under 'some' load.

[DavidColey]

Ok.  I wish I could be of more assistance.  I am not seeing an increase in RAM or CPU usage.  I do have many traditional map services running from the federated site, along with about 25 image services. Most of the traditional map services are running in the dynamic shared instance - of course that's not a web adaptor issue.... per say

[Scott_Tansley]

Thanks for sharing the info.  I found out the hard way on all of this within 24 hours of release. Pre 11.1 you could run a web server with 2 CPUs and 4GB of RAM with no issues.  With 11.1 and multiple app pools, I was seeing clients 'burst' up to 6GB's and steady state was about 4.5-5GB.  The 11.1 documentation does say that 8GB is required as a minimum now.  

Given your exceptionally large web server, I'm not surprised that you're not seeing change, other than the additional processes.  I have to say that I'm really pleased to see that there are different App Pools for each Web Adaptor, I think that's a great change from a security perspective.  

Thanks again for sharing, it all helps the community and getting early understanding of the new release.

[DavidColey]

Thanks @Scott_Tansley  - that is why I do it - to help the community and others avoid pain if possible and to just add to the overall understanding.  To me, this is the point of the geonet forum.  It's great when we can get answers but its just as important to understand what is working, not just what is failing. 

As far as our web server goes, I didn't really consider it 'exceptionally' large, but we do use it for hosting a lot of dev edition WABS, virtual directories, and setting up php to pass pictometry credentials, so we did build in some size.  BTW, I thought I read that of the 2 new required web components, one of them was not yet compatible for Windows Server 2022?  

[Scott_Tansley]

I hadn't seen that, so I need to do some research.  It would currently only affect one client.  Most of my clients came on at 10.7.1/10.8.1 and are on Server 2016 or 19.  

[cepsgis]

We too are having this issue. We have done the uninstall/ reinstall twice now. I'm digging through process logs to see if there is anything I can find that locks it up. Are you all running windows server 2016? I think that may be part of the issue.