ArcGIS Web Adaptor 11.1 App Pool freezes

[Scott_Tansley]

Hi.   

I've recently upgraded a client to ArcGIS 11.1, and we're having random problems with the new web adaptors.  I'm looking to see if anyone has observed the same issues.

So the clients ArcGIS Enterprise was deployed at 10.8.1.  There's an IIS Web Server in the DMZ.  A single host with the rest of the base deployment, which is only used for Hosted Feature Services.  WA's exist for portal and hosting.  There is a third machine with a general-purpose ArcGIS Server, federated and primarily serving Map Image Layers.  There is a Web Adaptor called server.

There have never been any repeated outage issues.  The environment was upgraded to 10.9.1 last year.  Once again no issues.

They were upgraded to 11.1 two weeks ago.  Immediately, we found that the machine was running out of memory, we noted the advice given in the new dependencies, and increased the RAM from 4 to 8GB.  It sits at around 5-6GB with no issues, and we have not seen any spike above 6GB to date.

After adding RAM it all seemed to settle for a few days.  But now, every couple of days the IIS application pool for the 'server' web adaptor will just stall.  IIS logs show 200/304 responses for everything up to the freeze/stall and 500 for everything.  There is nothing untoward in the requeste.

ArcGIS Server is still available on 6443 and can be accessed.  It just isn't receiving requests from IIS.  With Info logging turned on, it shows the last good 200 request from the WA.  Then nothing, no errors, no issues.  It's just as if it's sat there waiting for a request and not receiving it.

There have been no firewall or environmental changes recently, the only change is the upgrade to 11.1 and the addition of memory.

On the web server there is nothing in event viewer, system/admin/security or IIS application logs.

I'm blind.  It's just as if the App Pool WA says I've had enough.

The only way to bring the application back online is to restart IIS.  On the AppPool you can stop it.  But it will not start unless IIS is restarted. 

I'm currently blind.  We've external ping monitoring in place so we know when the healthCheck API fails, but there's nothing else we can do but monitor and restart at this point.

[RyanUthoff]

The issues we're experiencing have not changed between patch 1 and patch 2. I haven't noticed any difference at all. When the issue happens, Portal continues to operate as expected, but any feature services within the map viewer just stop loading. Restarting IIS immediately fixes the problem. I don't have enough IT knowledge to troubleshoot anything within IIS, so I'm not sure what to look for in there to narrow down the issue. But I did implement the workaround that Luke posted and so far, it's been alright (even if it is a temporary fix).

I do have a support case open with Esri support. So far, the only response from support is to implement the workaround I've already implemented and that they would keep me updated with any updates to the web adaptor. 

[JonEmch]

Hey Ryan, could you DM me your case number?

[Edgar_W_Iparraguirre]

You could also check <webadaptor>/rest/info, if it takes f.e. more than 30s it must be that the web adaptor has become unresponsive, and so the application pool / iis must restarted. 

rest/info should not take more than 500ms

[DavidColey]

You know, I outlined the specs of our web adaptor machine in this post several months ago.  We still have yet to experience any of the issues that the patches address.  Nevertheless, I of course have applied these patches.

For those that continue to experience "App Pool Freeze" issues, I haven't seen or read any specs on any of the machines that are failing.  I only mention this because of 3 items:

DMZ - we use F5. For us, our F5 does no load balancing, it merely directs requests.

OS - we are still at Windows Server 2016 on the server hosting our web adaptors.  One of the components required to run web adaptor at 11.1 is the Web Deploy 3.6 .  When you go to this download page, and review the system requirements for the Web Deploy 3.6, you can see that Windows Server 2022 is NOT listed among the supported operating systems:

• System Requirements

  Supported Operating Systems

  Windows XP, Windows 7 Professional, Windows Server 2003 Service Pack 2, Windows Server 2012 R2, Windows Server 2008 R2 SP1, Windows 8 Pro, Windows Server 2012, Windows Vista, Windows 8.1, Windows Server 2016, Windows Server 2019

  • The supported platform is 64-bit for this download.
  • The .NET 2.0 Framework SP1 or greater must be installed.

  Does this mean that the Web Deploy 3.6 supported on Windows Server 2022?  I don't know - Microsoft nor ESRI say anything about it.  NOR does the Microsoft documentation say anything about IIS 10

  For others, perhaps they did not follow this item from here:

  ArcGIS Web Adaptor 11.1 system requirements 

  Caution:

  If you installed the ASP.NET Core Runtime - Windows Hosting Bundle before enabling IIS and the required components, you must repair the bundle installation using the bundle installer.

  Also from the system requirements above:

  Memory - the system requirements dictate that the 'web adaptor' now requires 8Gb minimum:

  Hardware requirements

  ArcGIS Web Adaptor (IIS) requires a minimum of 8 GB of memory, with more potentially required depending on the number of ArcGIS Web Adaptor instances installed and the number and size of the requests received.

  Does this mean 8Gb per Web Adaptor?  I don't know.  We are running 3 Web Adaptors on a 16Gb Windows 2016 Server with 4 cores, 8 virtual.  So according to the hardware requirements, we are under-memory. 

  Yet we run about 20% memory utilization on the server day in and day out:

  

  So that's it.  I guess I "lucky?"  I don't know, but I do know more fine-grained info needs to be provided on this thread.  

[IanIce1]

- ArcGIS Enterprise scaled out across 3 Virtual Machines (all windows server 2019).

- Federated Hosting Server has 4 virtual cores (still had App Pool Freezes when upped to 16 cores). 24 GB Memory.

I installed the first patch with no luck. I'm not too optimistic about this 2nd patch. So far Luke Savages approach is holding about 98% of the time. I can still get the WA to **bleep** out occasionally but geometries don't just freeze and disappear, they still render, we just can't select records (map viewer/attribute table). These "downtimes" are pretty short lived.

Assuming the folks in here worked with ESRI on the issue....They know which environments are problematic. My rep had to close our case out b/c it was open too long and we're "stable" thanks to @LukeSavage .  The rep mentioned a 2nd patch was on the way and would notify me.....I was not notified. Not liking what I'm hearing from others, but glad to be part of the community.

[ChrisCarter3]

Just to add to the anecdotal evidence... Patch 2 has improved things slightly for us (on 2 portals) but not eliminated the problems with WA 11.1 entirely. We no longer have to restart the app pool / IIS, but we are still having the WA hang for 30s or so when too many tiles are requested from a cached map service. My support case was closed before Patch 2 came out because "the bug is fixed in 11.2", which doesn't help us much right now.

[JonEmch]

Hey @ChrisCarter3 ,

   Would you mind DM'img me your case number?

[BryanBoutz1]

I still had issues after applying the patch, but was able to resolve them by going into the advanced settings of the application pool and increasing the maximum queue length and number of worker instances.  Of course, this also requires more machine resources, but eventually we were able to find a balance that works well.

[KaitlynStevens]

Hi @BryanBoutz1 I'm sorry to hear you were still encountering issues after installing the patch. I understand you have a workaround, but if you're willing to look into this, please contact Support and message me your case number.

Thanks,

Kaitlyn

[KPattison]

We are also having the same issues that others are listing and they have improved a bit with the WA Patch 2, however we also have one extra symptom where if we make an item public sometimes it might still ask anonymous (not signed in) users for login and will do so for around 24~ hours before it stops asking for the login and realises it is public.

After looking at the IIS app pool I can see our recycling occurs every 1740 minutes or 29 hours which lines up with when out portal item will realise it is public and not ask anonymous users for login.