ArcGIS Server & webadaptor, registering every 15 mins

[LachlanWainwright]

Hi

We are running ArrcGIS Enterprise 10.9.1 on Windows and one of our ArcGIS Server sites is un-registering and registering the web adaptor every 15mins.  The setup has 3 WA's, Portal, ArcGIS and GIS (hosting server) and it's the hosting server WA that's throwing a shoe. Just the one, the others are rock solid.

ON a bad blowout you get the 500 error and a wait until the next cycle and it 'self rights'.

We have considered the following:

•   the https2 and disabled that on IIS, Didn't Fix It (DFI)
•   uninstall/install the WA, DFI
•   thought it might be ArcGIS monitor, stopped that completely including the thing you install on the machine, DFI

Can;t see anything in the logs for AGS, Portal, IIS or event Viewer.

We don't have any tasks running every 15 that could be blowing it out.

On a bad blow out it empties the config file likes the old 10.2 problem and then magically comes back on the next unreg/reg 15mins later.

Has anyone seen anything like this?

We do have a ticket with ESRI Australia and Adi is helping like a champ, just seeing if anyone else has seen this before.

Please & thanks

Lachlan

[David_McRitchie]

Quite a challenging sounding issue, you have probably already checked, but worth ensuring that the environment is fully patched, as there are a lot of patches available for 10.9.1

I would also want to rule out scaling, as we have seen web adaptors get overwhelmed by traffic. Are we certain this is happening on the dot at a 15 minute timestamp or is it happening roughly every 15 minutes? Do we see any request spikes when we look at the Server Manager Statistics page?

Are things working fine with CPU and Memory usage, with no reported spikes above 90%?

Are the Web Adaptors hosted on the same machine with the ArcGIS Server, or are they separate?

Hopefully those questions help narrow this down a bit further.

David

[LachlanWainwright]

Hi @David_McRitchie 

Thanks for responding 🙂

Every 15 minutes without fail, nearly to the second, for each and every hour of the day, 06, 21, 36, 51 past the hour.   4 CPU and 32GB RAM, no spikes above 90% or anywhere near 90%.

Installed on the machine:

Portal for ArcGIS & 3 WA's with 2 rock solid, haven't changed since the day they were setup.

Question about patching...let's say yes, minus a few from late last year....

The 2 AGS sites are one separate servers (2 machines per site)

WA's: 

    /portal   <-- rock solid

   /gis (hosting) <-- the one that's consistently failing

  /arcgis (federated)   <-- rock solid

The C:\inetpub\wwwroot\gis\webadaptor.config file gets rewritten and on the shared config (gisserver\config-store\web-adaptors\) the guid.json files are as well and many orphaned guid.json.rlock files.

Nothing in any logs (on verbose), other than it saying it's un-registering and registering the WA.
No services that spike and it appears this has been happening since install/setup of this one site.
No spikes on any of the machines anywhere CPU/RAM wise, nothing in event viewer.

It's like a processing is running and tripping it off just for giggles.
Annoyingly cannot find what it is 😫

Hope this helps & thanks for helping

[David_McRitchie]

Hey Lachlan,

A very challenging one, but agreed that is interesting it is occuring every 15minutes.

I'll be honest, I am not certain on this one, It is good you have raised a Support case on this. I did take a look against Esri UK cases and unfortunately did not see any similar behaviour logged before.

These suggestions might be helpful, or not of much use but the fact we see the WebAdaptor.config file emptied after a bad blow out has me wondering what could be corrupting the file.

If it is possible, I would consider double-checking certificates currently in use on ArcGIS Server, and IIS as well as the ArcGIS Server Service account.

Presuming this is similiar to what you mentioned used to be an issue at 10.2 (I will admit this is before my time with ArcGIS) perhaps the following forum sheds light on further ideas we can try. It mentions at 10.3 onwards that the ArcGIS Server checks the config-store for the presence of a file with GUID that matches that of the Web Adaptor. Perhaps there is a permissions issue with this, resulting in a corrupted Web Adaptor file.

For this running Configure ArcGIS Server Service utility will rule out a potential permissions problem.

Happy to keep discussing this, hopefully get to the bottom of this soon,

David

[LachlanWainwright · Accepted solution]

Hi @David_McRitchie 

We figured out what was blowing the web adaptor...Powershell!

One of my workmates searched good old AI and it suggested to use Microsoft Process Monitor, to see what was going on with the server, as soon as we did this it showed a sea of Powershell commands about the Web Adaptor getting registered, over and over again....

Backgound, we use DSC to do the installs.  Before you get going with DSC you set the ConfiguratoinMode to 'ApplyAndMonitor'.  Somehow the reverted back to the original setting, ApplyAndAutoCorrect which can cause things to run & run & run & run.

We applied the setting again and the WA immediately stopped registering.  It's been rock solid since Saturday 1622 Aussie/Strayla Time

Thanks for your help, appreciate you taking the time.

Lachlan

[JoshuaBixby]

Thanks for following up and sharing the root cause and solution.  This is the kind of small thing that can really trip people up, so getting it out in the web more broadly can only help.