ArcGIS Web Adaptor 11.1 App Pool freezes

[Scott_Tansley]

Hi.   

I've recently upgraded a client to ArcGIS 11.1, and we're having random problems with the new web adaptors.  I'm looking to see if anyone has observed the same issues.

So the clients ArcGIS Enterprise was deployed at 10.8.1.  There's an IIS Web Server in the DMZ.  A single host with the rest of the base deployment, which is only used for Hosted Feature Services.  WA's exist for portal and hosting.  There is a third machine with a general-purpose ArcGIS Server, federated and primarily serving Map Image Layers.  There is a Web Adaptor called server.

There have never been any repeated outage issues.  The environment was upgraded to 10.9.1 last year.  Once again no issues.

They were upgraded to 11.1 two weeks ago.  Immediately, we found that the machine was running out of memory, we noted the advice given in the new dependencies, and increased the RAM from 4 to 8GB.  It sits at around 5-6GB with no issues, and we have not seen any spike above 6GB to date.

After adding RAM it all seemed to settle for a few days.  But now, every couple of days the IIS application pool for the 'server' web adaptor will just stall.  IIS logs show 200/304 responses for everything up to the freeze/stall and 500 for everything.  There is nothing untoward in the requeste.

ArcGIS Server is still available on 6443 and can be accessed.  It just isn't receiving requests from IIS.  With Info logging turned on, it shows the last good 200 request from the WA.  Then nothing, no errors, no issues.  It's just as if it's sat there waiting for a request and not receiving it.

There have been no firewall or environmental changes recently, the only change is the upgrade to 11.1 and the addition of memory.

On the web server there is nothing in event viewer, system/admin/security or IIS application logs.

I'm blind.  It's just as if the App Pool WA says I've had enough.

The only way to bring the application back online is to restart IIS.  On the AppPool you can stop it.  But it will not start unless IIS is restarted. 

I'm currently blind.  We've external ping monitoring in place so we know when the healthCheck API fails, but there's nothing else we can do but monitor and restart at this point.

[IanIce1]

Hi Marshal,

We upgraded to .NET 7.0 SDK (v7.0.305). In addition, we followed the AppPool approach by @LukeSavage. Initially, I only did this for the AppPool 4.0 which didn't help, but now that it's configured on the enterpriseAppPool on both server and portal machines it seems to be holding up. Fingers crossed.

[IanIce1]

But you might want to first try sticking with the supported 6.x versions (our IT guys persuaded me to try 7.0) There was an update about 8 days ago (6.0.19). https://dotnet.microsoft.com/en-us/download/dotnet/6.0

[Scott_Tansley]

All, I didn't want to say until the patch was formally released, but I've been running it in the client's environment for 10 days.  I was privileged to be part of an early-release program.  We backed out @LukeSavage's workaround and implemented the patch on (NZ) Wednesday 21st.  The environment has been stable/reliable ever since. 

I have no issues with recommending its use based on my observations. 

I'll be back into the swing of upgrading production environments from this evening and implementing this patch as a part of the process.

It's been a valuable experience being supported by the Esri dev team on this, their communication and attitude has been amazing, and I've been thrilled to be small cog in a very big wheel that has brought us to the point of a reliability patch.  Thanks to everyone that has joined on to this discussion, providing weight and gravity to the issue, when I thought I was the only one.  I'd also like to thank @LukeSavage for the temporary workaround which got many of us out of a bind. 

Hopefully, we can now move on with our 11.1 journeys and reap the rewards of the upgrade.

[IanIce1]

Best of luck! As the patch did not work for us 😞

[Scott_Tansley]

I noted your comments above.  Is it IIS that is crashing or the server?  I wasn't clear.

[IanIce1]

Hi Scott,

It's web adaptor related. When our data freezes, I notice the ArcGIS Server and admin page (using web adaptor name) are hung up and do not load. However, our actual ArcGIS Server service and ArcSOCs are up and running just fine. We've been stable since I fully adjusted both of the AppPools on server and portal vm's. We also upgraded our .net core software (not sure if that was an issue to begin). 

[asergio]

It didn't work for us either, i will try to implement the @LukeSavage  workaround.

[KaitlynStevens]

Hi @asergio 

In speaking with the product team, we don't believe the workaround is applicable to the ArcGIS Web Adaptor (IIS) 11.1 because the 11.1 version utilizes .NET 6 (whereas earlier versions were utilizing .NET 4). Do you have an existing case number with Support I can reference? If not, please get in touch with Support so we can look into this further.

Thanks,

Kaitlyn

[asergio]

Hi @KaitlynStevens .

We implemented the @LukeSavage  workaround  this Friday late morning, after verifying that the patch did not solve the problem, and from that moment on no more problems occurred....."so far".

Let's wait and see how this week goes, if the problem of ArcGIS Server 11.1 via Web Adaptor IIS 11.1 becoming unresponsive for a few seconds, or even minutes, does not reappear again.

Thanks,

Antonio Sergio

[asergio]

As an update, it has been a week since we implemented the workaround and everything is still working and no more problems have occurred. It should be noted that the problems occurred even after the patch was installed.