topic Re: After upgrading 11.1 arcgis server is not responding in ArcGIS Enterprise Questions

After upgrading 11.1 arcgis server is not responding

AvinashDalvi_ — Tue, 02 May 2023 18:59:20 GMT

EC2 server where I upgrade its work there ArcGIS server but what I did I copy same machine and created another three machine everywhere facing issue. ArcGIS server is not responding. I tried restart and stop doesn't work. Also reauthorisation with new license 11.1 is completed successfully.

With same upgrade process ArcGIS portal and Datastore is working fine. Only issue ArcGIS server.

Error :

"Hmmm… can't reach this page"

Locahost refuse to connect

When I checked window-> services then ArcGIS server service is getting stopped continuously i tried to start bug again stopping. how to check what is issue ?

Re: After upgrading 11.1 arcgis server is not responding

MichelleCasey2 — Fri, 12 May 2023 16:03:49 GMT

Thanks for raising this query.

The are some important Considerations for upgrades when upgrading to ArcGIS Enterprise 11.1 such as .NET.

Please see some additional documentation here: Upgrade ArcGIS Enterprise which contains a link to Upgrade ArcGIS Server

Hope this helps.

Re: After upgrading 11.1 arcgis server is not responding

MichaelSnook — Fri, 19 May 2023 02:21:50 GMT

Hi Michelle

I currently have a ticket in process about a similar issue and it's been lingering for the last 2 weeks. I'm posting here to get as much help as I can before finally rolling back to a VM snapshot of 10.9.1. I 'upgraded' my production enterprise to 11.1 on 5/6. Since then ArcGIS server crashes/stops responding about 2-3 times a day. The only thing that seems to 'fix' the issue is rebooting the server. I have scoured ArcGIS server logs, windows server logs, and nothing seems to inidcate that a crash is coming or what causes it. There have been no changes to the server itself since the upgrade and I never had this issue when 10.9.1 was running. Please keep in mind that I have done everything in the documentation for the migration to 11 from 10.9.x. All SOEs have been upgraded, server hardware is at least min spec etc. Oddly, the installation/'upgrade' went without a hitch. Also, Portal and the Datastore all seem to be fine...it's just that Server stops responding altogether. I can't reach server manager or the rest endpoint either internally or externally. I'm basically having to watch my system all day to see if it's still running and reboot if it's dead so I can keep my production services up.

Please help.

Thanks.

Re: After upgrading 11.1 arcgis server is not responding

PDSGIS_Department — Fri, 19 May 2023 21:46:21 GMT

Same thing is happening with our installation.

Re: After upgrading 11.1 arcgis server is not responding

MichaelSnook — Mon, 22 May 2023 13:47:54 GMT

Do you have any ideas or suspicions to the cause? I've been trying to check every possible angle and when I fix something that I think is the culprit it crashes again.

Thanks!

Re: After upgrading 11.1 arcgis server is not responding

PDSGIS_Department — Sat, 20 May 2023 02:14:59 GMT

The server ran for days without issue in a test environment. It was only after we made it the production server that the issues started. One of the suggestions from tech support was about increasing the pagefile:
“ Check Page file size across the machines. Default is system managed which is usually a lot less than best practice. I would consider increasing the page file size to a minimum of 1.5X RAM but close to 3X RAM is best. “
We are testing this currently but still looking for other solutions.

Re: After upgrading 11.1 arcgis server is not responding

MichaelSnook — Sun, 21 May 2023 21:07:08 GMT

Testing this out over the weekend and (I don't want to jinx it but) I haven't seen a crash since I upped the pagefile size...it was rather small at 5GB. I bumped it to 16 and hoping that's a fix!

Re: After upgrading 11.1 arcgis server is not responding

MichaelSnook — Mon, 22 May 2023 14:32:15 GMT

Unfortunately it crashed again early Monday morning. Waiting for support again...

Re: After upgrading 11.1 arcgis server is not responding

PDSGIS_Department — Mon, 22 May 2023 14:52:59 GMT

Same here. Support is stumped. We're going to put our 10.9.1 server back into production.

Re: After upgrading 11.1 arcgis server is not responding

MichaelSnook — Wed, 31 May 2023 02:13:05 GMT

Still stumped here too...there is nothing in the logs, no error reports at all to even begin to diagnose. I upped the page file and still crashes. I removed SOEs, still crashes. At least now it's only once a day which leads me to believe theres a memory issue when the traffic spikes but my VM stats -- RAM/CPU/Network-- appear to be normal when the crashes happen...I will have to roll back to 10.9.1.

Re: After upgrading 11.1 arcgis server is not responding

MichaelSnook — Wed, 31 May 2023 19:34:52 GMT

Check this thread...there seems to be a common theme here:

https://community.esri.com/t5/arcgis-enterprise-questions/arcgis-web-adaptor-11-1-app-pool-freezes/td-p/1291473

Testing out the IIS config to see....

Re: After upgrading 11.1 arcgis server is not responding

PDSGIS_Department — Wed, 31 May 2023 19:44:27 GMT

It's worth a shot. Let us know if it works for you. We're taking a break from attempting to upgrade until next week.

Re: After upgrading 11.1 arcgis server is not responding

MichaelSnook — Fri, 02 Jun 2023 16:48:50 GMT

It definitely seems like that did the trick...3 days now without a freeze/crash. All systems and services seem to be operating normal.

Re: After upgrading 11.1 arcgis server is not responding

Ian_Ice — Thu, 29 Jun 2023 15:53:29 GMT

I wish I came across these threads earlier. Our organization just spun up Enterprise for the first time at 10.9.1. Everything was great! We had no users at the time (just test users). 11.0 came out and we figured we would upgrade since it was light weight with no real users. Everything was fine in 11.0. Well we went ahead and decided to upgrade to 11.1 since it was a "long term" release and had some new resources. We still had very few users on it. We upgraded and tested for 2 weeks before we deleted the VM snapshots. Shortly after, we started noticing that our data would freeze. At the same time, the ArcGIS Server Manager page wouldn't load, it would just sit there spinning and sometimes erroring out. It was a complete roller coaster. It could be down for seconds and sometimes minutes (22 minutes one time). this would happen only when we had maps and apps up with data. Portal/Portal admin and datastore software was fine and accessible, although when you clicked on a feature layer/map image to view it, there would be no data there and item details were inaccessible.

I still have a support ticket open. We've checked permissions across the 3 virtual machines/software, certificates, web adaptor pooling resources (which they had my increase some numbers), ruled out our DNS/aliasing as an issue. I had our IT guys throw more cores at it (testing) and it seemed to work for a week and came back just recently. I turned off all shared and dedicated instances on the server and still happened with only copied data.

Very frustrating to work with this new software. Unfortunately we have to downgrade to 11.0 and do not have snapshots anymore, just a portal backup file that was created during the upgrade process. Most of the time lost here is working with support to fix this issue and trying fixed ourselves. Very disappointed customer here.

Re: After upgrading 11.1 arcgis server is not responding

Ian_Ice — Fri, 30 Jun 2023 20:16:37 GMT

Update,

The new "Reliability" patch that esri released yesterday did not work for us. I was almost ready to pull the plug and attempt a downgrade until I revisited the IIS/Web adaptor AppPools.

It turns out that support only had me update one of the Web Adaptor AppPools on both the enterprise and portal machine, the v4.0 one. After changing the parameters to the enterpriseAppPool as described in the link that Michael Snook shared and updating our .NET core software (someone suggested) we are finally looking stable. We went back to 4 cores and started stress testing it hard and it's not breaking anymore. fingers crossed.

Hope this helps someone!

Re: After upgrading 11.1 arcgis server is not responding

GaiusK — Mon, 17 Jul 2023 15:50:47 GMT

@AvinashDalvi_ I have come across a similar issue, where the user upgraded to AE 11.1 and the GIS Server services intermittently goes down, some services do not appear on the Map Viewer and the Server Manager page is inaccessible with the Web Adaptor URL.

After researching, we could find a temporary workflow, from the IIS Manager reduce the recycle time to 5 mins in the "ArcGISWebAdapterAppPool" and increased the Queue Length of application pool from 1000 to 65535.

Further research on this same, we found that it is a BUG-000158629 and installing the ArcGIS Web Adaptor (IIS) 11.1 Reliability Patch resolve this issue.

Please let me know if this solution was helpful to you.