504 gateway timeout when printing through utility.arcgis.com

[JohnFannon]

Hi - we have a stand-alone ArcGIS Server (v11.5) that we are hosting for a customer to enable them to do custom prints from ArcGIS Online apps. The print service is set as synchronous (required for printing from Web App Builder apps) and secured with user credentials and these are embedded within an item in the customer's ArcGIS Online account and the item is then shared and used across ArcGIS Online (e.g. in the print widgets etc). This approach has been in place and working fine for a couple of years. 

For information - we cannot currently change the print service to asynchronous, because this mode is not supported by older apps (e.g. Web App Builder) that the customer is still using. 

Recently the customer has started experiencing failed prints due to intermittent 504 gateway timeouts. We have investigated and found that there are no timeouts happening on the ArcGIS Server (the timeouts are set quite long and the load is minimal) and when a print fails, there are no errors in ArcGIS Server and the output has been generated in the output directory. 

We have tested printing directly using the ArcGIS Server URL (user needs to enter credentials each time) and this consistently works without errors. However, we can't use this in production.

Our investigation suggests the timeout is coming from the utility.arcgis.com proxy that is used when using an item with embedded credentials:

The error in the browser also states "Error from CloudFront". CloudFront is not part of our infrastructure, so this error must therefore surely be coming from the utility service proxy. 

Further testing has revealed that successul prints tend to complete in <30s, whereas failed prints tend to fail around the 30s mark. I personally believe this indicates there is a timeout setting (possibly 30s) somewhere in the utility.arcgis.com proxy, probably in CloudFront. This would explain why longer prints would fail but there are no errors in ArcGIS Server, because the proxy is timing out before the request is completed in ArcGIS Server.

We have raised a support call but are not getting very far. Can anyone confirm if there is indeed a timeout setting for the utility service proxy if so, what it is set to?

[JohnFannon]

Just to follow up on this - we have raised this issue with Esri UK support, but so far they have suggested it may be raised as an "enhancement", despite the fact it was working for approximately 2 years previously and so it's quite likely that a change was made by Esri Inc that has caused it to start timing out after 30 seconds. 

They have also suggested creating an idea - which we have now done here:

https://community.esri.com/t5/arcgis-online-ideas/extend-the-timeout-on-the-utility-arcgis-com-proxy...

[JemGIS]

This is exactly what is happening to our secured map services, although the timeout for us is 5s. We have a stand-alone ArcGIS Server (v10.9.1). I have had a ticket open with ESRI (US) since October 30th and we have gotten no where.

We use this method to serve out utility data (sewer, water, etc) which needs to secured but accessible to our organization through AGOL. We have never had this problem, and it appears to have started after the last AGOL update in early October. These map services load perfectly fine when choosing to NOT store the credentials, but then the users are prompted for authentication every time the map service loads. We don't think there is a firewall issue or anything on our end. It appears to be an issue with utility.arcgis.com as you described.

Our workaround has been to add the map services as items in our Enterprise Portal (storing the credentials there) and then adding that Enterprise item as an item in our AGOL organization. If we didn't have an Enterprise Portal, we would be dead in the water.

The only other 'recent' reference to this issue I could find in the Community:
https://community.esri.com/t5/arcgis-online-questions/unable-to-store-credentials-with-secured-featu...

On page 2 of the comments, I tried the password technique that the commenter mentioned but that did not work.

I fear that we might be headed down the 'idea' road as well. 

[JohnFannon]

As covered here: 

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DownloadDistValuesOrigin.html#Dow...

The default response timeout for CloudFront appears to be 30 seconds:

I expect that is where the problem lies, but getting Esri to accept that and change it seems to be the main issue.