Intermittent 504 gateway timeouts with hosted feature layers

[JohnFannon]

Hi

We have a customer for whom we have published some data as an ArcGIS online hosted feature service. We have a process that applies changes to the hosted layers based on changes in their database, including adds, updates and deletes. The process is written in python and the code is sending http requests to the layer's REST endpoint (i.e. addFeatures, updateFeatures, deleteFeatures).

The code is working fine generally and has been for some months, but we are increasingly seeing intermittent errors in the log for our process that are related to http requests, especially Error 504 (Gateway Timeout). I've looked into error 504 and this suggests a timeout is occuring between internal servers in ArcGIS Online. The error seems to occur most frequently when updating a specific layer (using updateFeatures), which has approx. 4500 features, but occasionally occurs with update requests to other layers with fewer features (approx. 500). It is totally intermittent, so during one run one request might fail, but then the next will succeed.

We have put in place some work arounds, such as making a number of attempts at the request when it fails and limiting the number of features updated per request. However, we are still seeing some instances of error 504.

I'm just wondering if anyone else has experience similar when applying regular updates to an AGOL hosted feature service?

Regards

John

GIS4BUSINESS

[KellyGerrow]

Hey John,

I'd suggest getting in touch with technical support. They can help troubleshoot the specific error, but this shouldn't be something you need to work around depending on how your script is written.

Let me know if you need some help getting in contact with support.

-Kelly

[AndyWright]

Hi John,

We are seeing the exact same issues syncing within an ArcGIS Runtime app.  Our suspicion is it is something within the server or network architecture, but haven't been able to pinpoint it yet.  I was wondering if you ever got to the bottom of this and if you would be willing to share the answer if you did?

Thanks,

Andy

[JohnFannon]

Hi Andy

No - we didn't ever get to the bottom of this and have instead worked around these issues by repeating the request a number of times if unsuccessful. Generally this works, but we still get the occassional instance, perhaps once a week, where it still fails, even after several requests.

My feeling is also that it's something internal to ArcGIS Online, perhaps a load issue where one particular server or database does not respond within an internal timeout. I would have thought Esri would be monitoring these things within the platform, so must be aware that it's happening.

You could probably work around the issue in a similar way within ArcGIS Runtime, by re-trying failed requests.

Sorry I can't be of more help.

John

[AndyWright]

Ok, thanks for the response John.  If we happen to get to the bottom of this I'll post something here ...

[KellyGerrow]

Hi John,

Can you provide some additional details about how you are requesting the layer, how often, and what the request is? This shouldn't be intermittently occurring so is something that should be investigate by technical support. 

Are there any patterns that this follows like after re-writing a service? Can you confirm that this is a service that is hosted in ArcGIS Online? If this is continuing, please get in contact with Esri Support Contact Support  to look into the specific issue.

-Kelly

[JohnFannon]

Hi Kelly,

The requests are to a hosted feature service in ArcGIS Online, via the REST API, through python using urllib/urllib2. The requests are either addFeatures, updateFeatures or deleteFeatures and we have a setting that limits the number of features per request (currently set to 2000).

At the moment the script is run every 15 minutes and the work-arounds we have in terms of repeating requests are generally working. Recently we have not seen too many failures and any failed updates generally succeed in a subsequent run of the script.

It would still be interesting to know what the cause is. I suspect something internal to the platform is timing out when the request contains a large number of adds/updates or when there is heavy load at peak times. Though I've no conculsive evidence to support this or have much time to see if there are any patterns.

John

[WASHSECTOR]

Hi, What was the solution for this problem ?? I have the same problem. My script was running normally and when i have more data it stops at all. It always giving the error of HTTP Error 500: Internal ServerError and sometimes Error 504.

[PaulDavidson1]

We're fighting the same issues with the ArcGIS Python library.

Appears to be totally random when it occurs.

We can run our script for 24 hours without a problem or it can throw an exception in 5 minutes.

It's generally when trying to read or write to/from a hosted layer in the Data Store.

The data is being created by Collector and then we're reading the inspection records, determining the result of the inspection (the status, passed, failed, etc...) and writing that value back up to the Feature's (a Hydrant) record so that the map symbology will change in quasi-real-time.

My thought was the same as yours, use nested try:excepts to give a few shots connecting correctly.  At some point, toss a fatal exception and stop the process.  We're running the process on a 15 minute loop so if it fails, it will start again within 15 min at the most.

It's not at all uncommon to see issues like this in the web, right?  Lot of switches, etc... that messages have to go through and possibly get bumped around.  It would be nice to see the logic for dealing with these issues put into the core code so that we don't have to hack around it.

[KellyGerrow]

Hey Paul,

Can you confirm that this is using a hosted feature service in ArcGIS Online? What Process are you using to update the data? overwrite or append? With overwrite there is a small amount of down time while the publishing process completes. For this reason, I'd recommend using append if you are adding or updating records. If this continues, please get in contact with technical support.

Here are some helpful blogs about the append functionality.

What’s new with hosted feature layers: Updating data gets more flexible with append 

What’s new with hosted feature layers: Getting to know Append Part 1 (December 2017) 

https://www.esri.com/arcgis-blog/products/arcgis-online/uncategorized/whats-new-with-hosted-feature-... 

-Kelly