ArcGIS Web Adaptor 11.1 App Pool freezes

[Scott_Tansley]

Hi.   

I've recently upgraded a client to ArcGIS 11.1, and we're having random problems with the new web adaptors.  I'm looking to see if anyone has observed the same issues.

So the clients ArcGIS Enterprise was deployed at 10.8.1.  There's an IIS Web Server in the DMZ.  A single host with the rest of the base deployment, which is only used for Hosted Feature Services.  WA's exist for portal and hosting.  There is a third machine with a general-purpose ArcGIS Server, federated and primarily serving Map Image Layers.  There is a Web Adaptor called server.

There have never been any repeated outage issues.  The environment was upgraded to 10.9.1 last year.  Once again no issues.

They were upgraded to 11.1 two weeks ago.  Immediately, we found that the machine was running out of memory, we noted the advice given in the new dependencies, and increased the RAM from 4 to 8GB.  It sits at around 5-6GB with no issues, and we have not seen any spike above 6GB to date.

After adding RAM it all seemed to settle for a few days.  But now, every couple of days the IIS application pool for the 'server' web adaptor will just stall.  IIS logs show 200/304 responses for everything up to the freeze/stall and 500 for everything.  There is nothing untoward in the requeste.

ArcGIS Server is still available on 6443 and can be accessed.  It just isn't receiving requests from IIS.  With Info logging turned on, it shows the last good 200 request from the WA.  Then nothing, no errors, no issues.  It's just as if it's sat there waiting for a request and not receiving it.

There have been no firewall or environmental changes recently, the only change is the upgrade to 11.1 and the addition of memory.

On the web server there is nothing in event viewer, system/admin/security or IIS application logs.

I'm blind.  It's just as if the App Pool WA says I've had enough.

The only way to bring the application back online is to restart IIS.  On the AppPool you can stop it.  But it will not start unless IIS is restarted. 

I'm currently blind.  We've external ping monitoring in place so we know when the healthCheck API fails, but there's nothing else we can do but monitor and restart at this point.

[KaitlynStevens]

Hi @KPattison if you are still having issues described in this thread and the patch has not completely resolved these, please create a Support case and message me when you have the case number. Support will be able to help determine whether this other symptom is related, or if there's further troubleshooting required.

Thanks,

Kaitlyn

[KPattison]

Hey Kaitlyn,

The IIS app pool resource increase mentioned by LukeSavage + the patch looks to have fixed our issue.

Looks like before the fix any shared item wouldn't know it was shared until the IIS app pool recycled which occurred every 1740 minutes or 29 hours.

All good now.

Thanks,

Kurt

[eryildiz]

We are also facing the same issue. We have applied the second patch. Do we need to apply the first one as well? Web adaptor stops from time to time.

[KaitlynStevens]

Hi @eryildiz the ArcGIS Web Adaptor (IIS) 11.1 Reliability Update 2 Patch accumulates fixes included in the first patch, so no, you do not need to apply the first. Do you already have a Support case open? If not, please log one and message me the case number when you can.

Thanks,

Kaitlyn

[CesarAugustoVelandiaRincón]

We currently also have the same problem.
We upgraded from arcgis enterprise 10.9.1 to 11.1 and ArcGIS Web Adapter (IIS) 11.1 has constant flashes even though we installed reliability patch 2 for ArcGIS WebAdaptor 11.1 (https://support.esri.com/en -us/patches-updates/2023/arcgis-web-adaptor-iis-11-1-reliability-update-2-patch). Even so with the installation of the patch the same disconnection is presented. The only thing that has helped us to correct the intermittency a bit was the implementation made by @LukeSavage and also mentioned by ESRI support in the open cases but it continues to present drops but not so continuous anymore.

[IanIce1]

Okay, I finished setting up our 11.1 development environment and published many datasets to it. Brought a few datasets in Map Viewer, zoomed in and out and the web adaptor immediately froze. The web map was hardly usable. In contrast to our production environment, this environment is a single machine deployment (Virtual Server), Windows Server 2019, 24 GB memory, 4 virtual cores. I used the builder to install the software. At this point I did not implement LukeSavages App Pool tweaks in this environment.

The first reliability patch did nothing for us in our scaled-out production environment (WS 2019). So, in this new dev environment, I went ahead and installed the 2nd reliability patch and we still had Web Adaptor Freezes. Just as bad as before the install. Went in IIS Manager and implemented LukeSavages tweaks and bam....we have some stability!

In both of our ArcGIS Enterprise 11.1 environments, LukeSavages approach is the only thing making this software stable. The patches did nothing unfortunately.

[eryildiz]

I have an update. This patch does not apply if you have multiple bindings on iis. We tried to install the patch multiple times and the dll's in inetpub never changed. We could fix that once we deleted one of the bindings. Now we are monitoring it. Please check the esri dll dates under c:\inetpub\wwwroot\server folder. If it is older than august , then the patch did not apply. Hope that helps for someone else.

[BryanBoutz1]

Thank you for posting. This is a VERY important detail.  In my Esri support call they had me look at the DLL dates and version numbers, but didn't comment as to if it was what they expected from the patch or not.  Given we didn't see much change from applying the patch, I am suspicious.  I wonder if we can get a comment from Esri as to what the correct DLL version and date should be?  I suspect we are in for a patch 3 where the installer handles the situation better or at least provides better instructions.

[eryildiz]

Our DLL dates were first of march or something like that. Now it is August. I don't know the versions though.

[JasonHarris2]

This was great info.  I checked all the sites that I had 'patched' but were still not working properly. In these cases, the associated DLLs in the WA directories had NOT been updated.  Still showed dates in March.  I tried to reinstall the patch on these systems, still no changes.  Removed 80 as a binding, installed patch, still no update.  Then, stopped the WWW Publishing via the service, which of course shuts down IIS, installed the patch and pow, updated DLL dates/version.  For good measure, I did a load test and tossed about 10 concurrent users at it for about 15 minutes, and it passed with flying colors.  Dare I say, this patch is installed and working now!