topic Re: ArcGIS Web Adaptor 11.1 App Pool freezes in ArcGIS Enterprise Questions

ArcGIS Web Adaptor 11.1 App Pool freezes

Scott_Tansley — Mon, 22 May 2023 05:04:41 GMT

Hi.

I've recently upgraded a client to ArcGIS 11.1, and we're having random problems with the new web adaptors. I'm looking to see if anyone has observed the same issues.

So the clients ArcGIS Enterprise was deployed at 10.8.1. There's an IIS Web Server in the DMZ. A single host with the rest of the base deployment, which is only used for Hosted Feature Services. WA's exist for portal and hosting. There is a third machine with a general-purpose ArcGIS Server, federated and primarily serving Map Image Layers. There is a Web Adaptor called server.

There have never been any repeated outage issues. The environment was upgraded to 10.9.1 last year. Once again no issues.

They were upgraded to 11.1 two weeks ago. Immediately, we found that the machine was running out of memory, we noted the advice given in the new dependencies, and increased the RAM from 4 to 8GB. It sits at around 5-6GB with no issues, and we have not seen any spike above 6GB to date.

After adding RAM it all seemed to settle for a few days. But now, every couple of days the IIS application pool for the 'server' web adaptor will just stall. IIS logs show 200/304 responses for everything up to the freeze/stall and 500 for everything. There is nothing untoward in the requeste.

ArcGIS Server is still available on 6443 and can be accessed. It just isn't receiving requests from IIS. With Info logging turned on, it shows the last good 200 request from the WA. Then nothing, no errors, no issues. It's just as if it's sat there waiting for a request and not receiving it.

There have been no firewall or environmental changes recently, the only change is the upgrade to 11.1 and the addition of memory.

On the web server there is nothing in event viewer, system/admin/security or IIS application logs.

I'm blind. It's just as if the App Pool WA says I've had enough.

The only way to bring the application back online is to restart IIS. On the AppPool you can stop it. But it will not start unless IIS is restarted.

I'm currently blind. We've external ping monitoring in place so we know when the healthCheck API fails, but there's nothing else we can do but monitor and restart at this point.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

DorothyMaguli — Mon, 22 May 2023 05:31:59 GMT

I as well have the same issues with web adaptors at 11.1, the outgates are
so severe.Everytime i do an installation and re installation as well. Any
help accorded to this would of significance

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

Scott_Tansley — Mon, 22 May 2023 05:38:16 GMT

Thanks for confirming I'm not alone. I have asked my client to raise this as an official support ticket. We have also tried an uninstall/install and that made no difference. Only restarting IIS is able to temporarily solve this.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

DavidColey — Mon, 22 May 2023 17:40:58 GMT

I have just completed my upgrade from 11.0 / 10.9.1 to 11.1 and can tell you that through Monitor I can now see we have increased the w3wp process count on the web server from 2 to 5.

We have a distributed deployment on Windows Sever 2016 VMs all 8-core, 32Gb, with the exception of the web server which is at 8-core 16Gb. The web server currently hosts 3 web adaptors for Portal, Hosting Server Site(2-machine), and a Federated Server Site(2-machine).

I am assuming at 11.1 the additional components for web adaptor - the hosting bundle and web deploy 3.6 are spinning up w3wp processes but I don't know. I completed the upgrade Saturday 5/21 9 pm EDT, so today is the first weekday under load.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

Scott_Tansley — Mon, 22 May 2023 19:50:02 GMT

Hey David yes. All web adaptors (prior 10 11.1) ran in a single app pool. As per Microsoft best practice, each web adaptor is now security isolated in it's own app pool, which increases the RAM required. I was aware of this. In our situation it's just one App Pool that freezes, which is the traditional Map Service Server. It seems to be when it's under 'some' load.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

DavidColey — Mon, 22 May 2023 21:01:46 GMT

Ok. I wish I could be of more assistance. I am not seeing an increase in RAM or CPU usage. I do have many traditional map services running from the federated site, along with about 25 image services. Most of the traditional map services are running in the dynamic shared instance - of course that's not a web adaptor issue.... per say

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

Scott_Tansley — Tue, 23 May 2023 00:42:10 GMT

Thanks for sharing the info. I found out the hard way on all of this within 24 hours of release. Pre 11.1 you could run a web server with 2 CPUs and 4GB of RAM with no issues. With 11.1 and multiple app pools, I was seeing clients 'burst' up to 6GB's and steady state was about 4.5-5GB. The 11.1 documentation does say that 8GB is required as a minimum now.

Given your exceptionally large web server, I'm not surprised that you're not seeing change, other than the additional processes. I have to say that I'm really pleased to see that there are different App Pools for each Web Adaptor, I think that's a great change from a security perspective.

Thanks again for sharing, it all helps the community and getting early understanding of the new release.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

DavidColey — Tue, 23 May 2023 13:28:24 GMT

Thanks @Scott_Tansley - that is why I do it - to help the community and others avoid pain if possible and to just add to the overall understanding. To me, this is the point of the geonet forum. It's great when we can get answers but its just as important to understand what is working, not just what is failing.

As far as our web server goes, I didn't really consider it 'exceptionally' large, but we do use it for hosting a lot of dev edition WABS, virtual directories, and setting up php to pass pictometry credentials, so we did build in some size. BTW, I thought I read that of the 2 new required web components, one of them was not yet compatible for Windows Server 2022?

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

Scott_Tansley — Tue, 23 May 2023 21:49:28 GMT

I hadn't seen that, so I need to do some research. It would currently only affect one client. Most of my clients came on at 10.7.1/10.8.1 and are on Server 2016 or 19.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

cepsgis — Wed, 24 May 2023 17:08:32 GMT

We too are having this issue. We have done the uninstall/ reinstall twice now. I'm digging through process logs to see if there is anything I can find that locks it up. Are you all running windows server 2016? I think that may be part of the issue.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

JonathanPollack — Wed, 24 May 2023 17:11:53 GMT

We also are having this issue. WebAdapter seemingly randomly craps out. We are able to just restart/recycle the AppPool to get things working again but this is a huge pain/issue.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

Scott_Tansley — Wed, 24 May 2023 22:08:21 GMT

Thanks for this. It's Win Server 2019.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

Scott_Tansley — Wed, 24 May 2023 22:09:16 GMT

thanks for confirming. I can confirm that a support ticket has been raised with our local distributor. It's early days, but it's good (bad?) to know we're not alone with this.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

DavidColey — Fri, 26 May 2023 12:53:18 GMT

We are not experiencing this on Windows Server 2016.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

JasonHarris2 — Tue, 30 May 2023 16:23:09 GMT

Same here. Win 2019. Only fix is app pool recycle. Happening on 3 systems that were just upgraded. We've had a ticket open with Esri, but havent gotten anywhere. Anyone here make any progress?

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

SavageTrimble — Tue, 30 May 2023 17:41:57 GMT

Fixed it as a workaround. For which config change that solved it, I have no idea but maybe this will help the community.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

Scott_Tansley — Tue, 30 May 2023 20:02:16 GMT

Thanks for sharing see workaround below. Seems a logical approach until we hear from Esri.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

Scott_Tansley — Tue, 30 May 2023 20:09:45 GMT

That looks a good approach. I like it.

Out of interest, did you try increasing the instances without the queue length? Or did you just go straight to belt and braces?

I’m assuming it eats memory with more instances?

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

SavageTrimble — Tue, 30 May 2023 21:31:31 GMT

Didn't see too much of a hit. Honestly, the server behaved better. And yes, I just updated both based on Microsoft articles about appPool instances for .net core and queue length for multiple requests.

Re: ArcGIS Web Adaptor 11.1 App Pool freezes

JasonHarris2 — Wed, 31 May 2023 17:30:33 GMT

So far, that fix is holding! Applied to 3 systems that where experiencing the issue and we've been up for 24 hours without incident.