502 Bad Gateway error appears intermittently ArcGIS Enterprise by Cloud Builder Azure

[AhmedShehata3]

Hello,

I'm having a very serious and annoying issue with ArcGIS Enterpise installed by Azure Cloud Builder.

Issue:

The error 502 Bad Gateway appears very frequently and in random times and the Portal becomes inaccessible. The issue appeared a few months 10.8.1 Base Deployment was installed, but after upgrade to 10.9.1 it still persists.

Behaviour and workarounds:

Sometimes the portal goes up again without interference and sometimes I have to restart the Portal Windows service or the entire Portal machine. Sometimes it lasts stable for weeks and all of a suddent it keeps failing several times in the day. Usually, the server manager URL keeps working normally, so it seems an issue with the Portal app/machine.

Troubleshooting:

I usually check the Event Viewer and can't find any lead. Same for the Portal Admin/machine logs. I used a SSL checked, but the certificate chain seems fine.

I have no idea what could cause this unstable behaviour and don't know where to start investigating. Does anyone have any idea?

Thanks

[Anonymous User]

Are you sure your issue resides with the Portal for ArcGIS? I would check your IIS logs this could be an issue with the ArcGIS Web Adaptor.

[AhmedShehata3]

The deployment was made by ArcGIS Cloud Builder for Azure, so it relies on Azure Application Gateway and there's no ArcGIS Web Adaptor component installed.

[Anonymous User]

Does you Portal home page return a 502 error?

Not having a web adaptor is an important clue. Perhaps, the gateway is receiving http requests and not trafficking them to the portal? I have worked with f5 in the past but I always integrate web adaptor into the architecture to hide portal address. Not saying this is the way to do it. It is just the way I have deployed in the past.

It may also be related to unsecure ArcGIS Server services that the portal is trying to display. You may need to store a credential with the service(s) portal is trying to display.

Sometimes these big cloud providers can 'spit the bit' as well.

https://status.azure.com/en-us/status

I would focus on the ArcGIS Server services and insure they are all published with https and have stored credentials.

[AhmedShehata3]

Yes, my portal home/admin page return 502 error when accessed via the public URL https://myorg.com/portal/home either externally and internally on the VMs . However, when I remote-access the VMs and try to access the portal home/admin page using the internal URL https://portalvm.internal.cloudapp.net:7443/arcgis/home it's accessible, but of course the content doesn't display and give data error. 

I'm suspecting the external content imported to the portal, but the items are numerous so I find it difficult to monitor them all. Will do a focused check on them now. 

[NicholasSadowy]

I think we have a VERY similar problem that I am trying to resolve:

ArcGIS Server (standalone) Site (2 servers in site) 10.9.1. Both server are running on Ubuntu linux 20.04. I don't have a portal (because it is standalone) and I AM using Microsoft's Application Gateway.

Our issue appears as a 504 Gateway Timeout when I attempt to access the Export Web Map Task in the printing tools through the public url / ip through the Application Gateway. If I instead go directly to one of the ArcGIS Server's port 6443 service port it works 100% of the time.

I have a ticket open w/ ESRI. Because it works at the port but not through theApplication Gateway, they (ESRI) said that it was an Application Gateway configuration problem. I have a ticket open with Microsoft, but I have reviewed the A.G. config in depth and haven't found any config issues. I'll post any updates when I make progress. 

[AhmedShehata3]

Thanks for sharing Nicholas! Have you installed the Standalone ArcGIS Server instances using Azure Cloud Builder?

[NicholasSadowy]

No, we actually spun up the linux servers, and installed manually.

[FraserHand]

Hi there,

We had a very similar issue with 502 being returned from portal. Is your 502 at around 60 secs from the request? This is the timeout on tomcat and we saw portal calling the sharing API multiple times in the Catalina logs before returning a 502. We mostly saw it in the sign in redirect. We also logged a call with Esri and got the response that app gw isn't supported. Can I ask what region you deployed to? We deployed everything in West US, and as a workaround we deployed a new app gw to East US with the GIS remaining in West US and everything worked as expected. We are still trying to track it down. If you connect then I'll let you know if we find a solution.

[NicholasSadowy]

We are deployed in South Central US. Our timeout happens at about 20 seconds.