'Internal server error 500' error message in Portal for ArcGIS

[JuhaHaanpera]

Hello hello,

The Portal for ArcGIS occasionally gives end users an 'Internal server error 500' error message, the root cause of which we have been hunting for a long time with ArcGIS Monitor, among other things, and searching the server logs without success. Have others encountered the 'Internal server error 500' error and how have you fixed the problem?

Our ArcGIS Enterprise environment, which runs on version 10.81, consists of a separate IIS Webadaptor Windows server on the DMZ, a Portal, and three robust GIS servers, and an MS SQL Geodatabase server. Attached is a simplified topology image of the Enterprise environment. The portal has hundreds of simultaneous end users on the browser and Field Maps from the public Internet, as well as twenty ArcGIS Pro users in the office network. Where else could we try to find the cause of the fault? 
I have run all the patches with the Esri Updater tool and all the updates provided by Microsoft are automatically installed on the servers.

Thanks in advance for the tips.

B.r. Juha

[Oiligriv]

try applying this Patch

https://support.esri.com/en/Products/Enterprise/arcgis-web-adaptor/arcgis-web-adaptor-iis/10-8-1#ove...

[JuhaHaanpera]

This patch seemed in advance to fix the 'Internal server error 500' error message. I installed it early last month, but unfortunately, end-users still occasionally get that error message. I have now further verified that the patch is installed nicely and Esri's Patchfinder finds it on the Webadaptor server.

We still have to continue hunting for the root cause of the error message and fix it. Good repair tips are more than welcome 🙂 

 

 

[JuhaHaanpera]

We looked through the logs of the Enterprise servers last week. It appears that when an IIS server issues an 'Internal server error 500' message to end-users, requests do not come from the Web Adapter to the GIS servers, but for some reason remain along the way or freeze on the IIS server. How would you try to fix the error? Would reinstalling the IIS server and/or Web adapter help?

[JuhaHaanpera]

We also activate the Web Adaptor's logging on, but it seems a bit limited in this case.

It writes some 5 MB sometimes after a boot and restart of IIS, or something like that and delayed or something like flushing to file seems a bit random.

Also, we're afraid that it's almost impossible to catch the verbose log for the error, as there are tens of thousands of requests per hour, the log for a day would be like many gigabytes.

The most interesting 500 status responses come from the map service export method.

 We noticed that some invalid requests give a 500 request for non-existing content, which should be 404.

like for request "https://<domain>/ <portal webadaptor>/nonexisting"
https://aineistot.esri.fi/portal/nonexists

Hunting continues for the root cause of the 'Internal server error 500' error.

[PatrickChase]

We had installed the stability patch and were good for awhile but then the 500 error just started up again. Have you discovered anything more about the issue?

[DarinaTchountcheva]

Hi Patrick,

Have you figured this out?

We just started experiencing the 500 error on Chrome Version 102.0.5005.63 (the latest one). We have Portal for ArcGIS site and we provide PDF files on this site. PDF files are a part of the Portal content. Some users get the 500 error and some don't. 

After looking at the web logs I found the following: 

When the PDF file is requested there is status is 200 (you can see the browser spinning trying to get the content), then immediately the log shows status 401 for the same file, and then error 500. 

PDF files are shared with everyone. 

Clearing the browser's cache and cookies do not resolve the issue. 

There are no issues on Edge or IE. 

I don't seem to have this issue with services published to the host server. 

[PatrickChase]

Hi Darina - sorry, nothing so far though when we get the 500 error it is not browser specific. The 500 error can happen through any browser client (Chrome, Edge, IE11).

[iamademyuksel]

   We encountered this issue after ram memory upgrade. While restarting the server after the ram upgrade, we also did a windows update at the same time. Server Manager was running, the portal service was starting, and a login page in an application I wrote specifically to log in from the portal was returning tokens when I tried to log in. When we typed https://domain.com/portal , it was returning "500 - Internal server" error even if it redirected to /home.

1. I rolled back the updates. Not resolved.
2. I did the Portal for ArcGIS updates but the problem was not resolved.
3. I reconfigured webadaptor.config and other IIS response headers, but it still didn't work.

  💡 We have been dealing with this issue for about 4 days. We noticed that the file "server.xml" located under the file path "C:\Program Files\ArcGIS\DataStore\framework\runtime\tomcat\conf" is empty. We rearranged it by taking as an example the one in our different Enterprise configuration. Maybe it wouldn't be right for me to share it, but I found a repostory on GitHub. I am sharing it below.

   ▶️ GitHub; https://github.com/Esri/arcgis-cookbook/blob/master/cookbooks/esri-tomcat/templates/default/server.x...

   If this is the solution for you, if you respond positively, you can help us contribute more to the development of the ArcGIS product family. 🤗

Best regards...

Adem Yuksel

[KarenTHChen]

I encountered a similar issue when I used Jupyter and ArcPy. Not sure what was the trigger but one day it showed "500 : Internal Server Error" when I opened the Jupyter browser with ArcGIS Pro. The problem continued after restarting the computer.

Finally, I solved the issue by deleting the arcgispro-py-clone environment and cloning it again, which can be done at ArcGIS Pro -> Settings -> Python -> Manage Environments.

Hope this helps some users.