topic Re: Intermittent 504 gateway timeouts with hosted feature layers in ArcGIS Online Questions

Intermittent 504 gateway timeouts with hosted feature layers

JohnFannon — Thu, 07 Dec 2017 16:02:25 GMT

We have a customer for whom we have published some data as an ArcGIS online hosted feature service. We have a process that applies changes to the hosted layers based on changes in their database, including adds, updates and deletes. The process is written in python and the code is sending http requests to the layer's REST endpoint (i.e. addFeatures, updateFeatures, deleteFeatures).

The code is working fine generally and has been for some months, but we are increasingly seeing intermittent errors in the log for our process that are related to http requests, especially Error 504 (Gateway Timeout). I've looked into error 504 and this suggests a timeout is occuring between internal servers in ArcGIS Online. The error seems to occur most frequently when updating a specific layer (using updateFeatures), which has approx. 4500 features, but occasionally occurs with update requests to other layers with fewer features (approx. 500). It is totally intermittent, so during one run one request might fail, but then the next will succeed.

We have put in place some work arounds, such as making a number of attempts at the request when it fails and limiting the number of features updated per request. However, we are still seeing some instances of error 504.

I'm just wondering if anyone else has experience similar when applying regular updates to an AGOL hosted feature service?

Regards

John

GIS4BUSINESS

Re: Intermittent 504 gateway timeouts with hosted feature layers

KellyGerrow — Thu, 07 Dec 2017 16:51:38 GMT

Hey John,

I'd suggest getting in touch with technical support. They can help troubleshoot the specific error, but this shouldn't be something you need to work around depending on how your script is written.

Let me know if you need some help getting in contact with support.

-Kelly

Re: Intermittent 504 gateway timeouts with hosted feature layers

AndyWright — Tue, 06 Mar 2018 15:07:28 GMT

Hi John,

We are seeing the exact same issues syncing within an ArcGIS Runtime app. Our suspicion is it is something within the server or network architecture, but haven't been able to pinpoint it yet. I was wondering if you ever got to the bottom of this and if you would be willing to share the answer if you did?

Thanks,

Andy

Re: Intermittent 504 gateway timeouts with hosted feature layers

JohnFannon — Tue, 06 Mar 2018 17:03:35 GMT

Hi Andy

No - we didn't ever get to the bottom of this and have instead worked around these issues by repeating the request a number of times if unsuccessful. Generally this works, but we still get the occassional instance, perhaps once a week, where it still fails, even after several requests.

My feeling is also that it's something internal to ArcGIS Online, perhaps a load issue where one particular server or database does not respond within an internal timeout. I would have thought Esri would be monitoring these things within the platform, so must be aware that it's happening.

You could probably work around the issue in a similar way within ArcGIS Runtime, by re-trying failed requests.

Sorry I can't be of more help.

John

Re: Intermittent 504 gateway timeouts with hosted feature layers

AndyWright — Tue, 06 Mar 2018 19:55:13 GMT

Ok, thanks for the response John. If we happen to get to the bottom of this I'll post something here ...

Re: Intermittent 504 gateway timeouts with hosted feature layers

PaulDavidson1 — Tue, 29 May 2018 07:13:48 GMT

We're fighting the same issues with the ArcGIS Python library.

Appears to be totally random when it occurs.

We can run our script for 24 hours without a problem or it can throw an exception in 5 minutes.

It's generally when trying to read or write to/from a hosted layer in the Data Store.

The data is being created by Collector and then we're reading the inspection records, determining the result of the inspection (the status, passed, failed, etc...) and writing that value back up to the Feature's (a Hydrant) record so that the map symbology will change in quasi-real-time.

My thought was the same as yours, use nested try:excepts to give a few shots connecting correctly. At some point, toss a fatal exception and stop the process. We're running the process on a 15 minute loop so if it fails, it will start again within 15 min at the most.

It's not at all uncommon to see issues like this in the web, right? Lot of switches, etc... that messages have to go through and possibly get bumped around. It would be nice to see the logic for dealing with these issues put into the core code so that we don't have to hack around it.

Re: Intermittent 504 gateway timeouts with hosted feature layers

KellyGerrow — Thu, 31 May 2018 22:19:38 GMT

Hi John,

Can you provide some additional details about how you are requesting the layer, how often, and what the request is? This shouldn't be intermittently occurring so is something that should be investigate by technical support.

Are there any patterns that this follows like after re-writing a service? Can you confirm that this is a service that is hosted in ArcGIS Online? If this is continuing, please get in contact with Esri Support Contact Support to look into the specific issue.

-Kelly

Re: Intermittent 504 gateway timeouts with hosted feature layers

KellyGerrow — Thu, 31 May 2018 22:24:13 GMT

Hey Paul,

Can you confirm that this is using a hosted feature service in ArcGIS Online? What Process are you using to update the data? overwrite or append? With overwrite there is a small amount of down time while the publishing process completes. For this reason, I'd recommend using append if you are adding or updating records. If this continues, please get in contact with technical support.

Here are some helpful blogs about the append functionality.

What’s new with hosted feature layers: Updating data gets more flexible with append

What’s new with hosted feature layers: Getting to know Append Part 1 (December 2017)

https://www.esri.com/arcgis-blog/products/arcgis-online/uncategorized/whats-new-with-hosted-feature-layers-getting-to-kn…

-Kelly

Re: Intermittent 504 gateway timeouts with hosted feature layers

JohnFannon — Fri, 01 Jun 2018 11:18:28 GMT

Hi Kelly,

The requests are to a hosted feature service in ArcGIS Online, via the REST API, through python using urllib/urllib2. The requests are either addFeatures, updateFeatures or deleteFeatures and we have a setting that limits the number of features per request (currently set to 2000).

At the moment the script is run every 15 minutes and the work-arounds we have in terms of repeating requests are generally working. Recently we have not seen too many failures and any failed updates generally succeed in a subsequent run of the script.

It would still be interesting to know what the cause is. I suspect something internal to the platform is timing out when the request contains a large number of adds/updates or when there is heavy load at peak times. Though I've no conculsive evidence to support this or have much time to see if there are any patterns.

John

Re: Intermittent 504 gateway timeouts with hosted feature layers

ChuckBenton — Fri, 01 Jun 2018 13:44:21 GMT

I'm seeing similar issues with timeouts when appending or deleting content from a feature layer that's hosted on ArcGIS Online. While not 100% consistent, it only happens with large files. The feature layer I'm working with is quite large, over 40M parcels, so the work being performed takes a while, which is to be expected and acceptable. I've looked for a timeout value that I can set to accommodate this without luck. Any suggestions are most welcome!

Re: Intermittent 504 gateway timeouts with hosted feature layers

PaulDavidson1 — Wed, 06 Jun 2018 23:47:42 GMT

Hi Kelly:

Yes, I can confirm that we're reading and writing to (2) hosted feature services in AGOL that are part of a Hosted Feature Layer. They're part of a Relationship Class of a 1:many relationship.

This is basically a Hydrant Inspection program.

We are using a Python script.

My solution is to loop the script 7 times.

With each loop, read an inspection table, determine an inspection status for new inspections since last loop, write status to the inspection table (the many) and to the Feature Class hosted layer record for that asset (the 1 of 1:many). This can write a multiple number of records at once.

Then sleep for two minutes while more inspections occur.

After 7 loops, it dies out.

Task Scheduler is used to kick it off every 15 minutes so if it does crash, we're only down with our quasi-real time map for ~15 mins max.

This process is working well but of course feels like a hack but it works so I'm good with it.

Looks to me like we're using edit_features rather than append:

# here is where we update the two tables
    if len(InspectionEdits) > 0:
        EditInspectionResult = FeatureTable.edit_features(updates=InspectionEdits)
        EditHydrantResult = FeatureLayer.edit_features(updates=HydrantEdits)

I'll read up on using append.

Not exactly intuitive to do an append for an edit but if that's BP, we'll go that route.

I have found that we sometimes crash just reading the Feature Layer info, it's pretty random.

It's the web is how I tend to look at it.

Thanks for the append pointer. I'll report back after a chance to work it over.

Could be a while since we're cranking along (it ain't broke...)

Re: Intermittent 504 gateway timeouts with hosted feature layers

PaulDavidson1 — Thu, 07 Jun 2018 00:07:12 GMT

Hey Kelly:

Looks like the postings are really just about using the append tool with the hosted layers. Our issue is crashes in code which I'm pretty sure is due to timeouts. Like Chuck says below, I haven't found anyway to set those values.

As Paul Barker lays out in his writings, we're basically doing upserts. I think most updates in a database work as upserts.

So I would think append is not applicable here but I'll dig into the ArcGIS python library when I can.

Re: Intermittent 504 gateway timeouts with hosted feature layers

JohnFannon — Thu, 07 Jun 2018 08:49:37 GMT

Our issues didn't result from using append. We are sending requests directly to the REST endpoint via python (as the code was developed prior to the ArcGIS Python API) and getting the 504 gateway timeout errors. Any methods you are using via the ArcGIS Python API will ultimately result in a http request to the REST endpoint of the service using applyEdits, updateFeatures, addFeatures, or deleteFeatures operations, which are exactly the same operations we are using directly via python.

Also, setting any timout options on the client-side requests makes no difference, since these cover the client-side timeouts only and the error (504 gateway timeout) comes from an internal timeout on the server-side.

Within our code, we are capturing the http error and re-sending the same request again multiple times. In most cases this works, but occassionally we still get failures after multiple attempts. These don't result in crashes as we just log the error and continue.

Re: Intermittent 504 gateway timeouts with hosted feature layers

PhilLarkin1 — Sat, 11 Dec 2021 13:25:27 GMT

I am also experiencing this behavior. After 4 months of flawless performance, an overwrite function has started failing intermittently due to a timeout. I'm using the python API (1.4.1) as show below. These are the hosted feature datasets I am updating:
http://spokanecounty.maps.arcgis.com/home/item.html?id=5805ca391e5f4c1ca6f0e58aee0d46a6
http://spokanecounty.maps.arcgis.com/home/item.html?id=b6b1e7f8e12c4a9ba78af72076aee7af
http://spokanecounty.maps.arcgis.com/home/item.html?id=848144f660a741f4b274360d494f15a1

I believe BUG-000115253 may be associated. This was identified in 1.4.2.

Bug synopsis: When using the overwrite method from arcgis.features.managers.FeatureLayerCollectionManager, ArcGIS API for Python returns the error "Exception: Job Failed.''

example:

import arcgis

gis = arcgis.gis.GIS("https://spokanecounty.maps.arcgis.com", <usr>, <pw>)

item = gis.content.get(itemID)

dataPath = "\\\\<path>\\Parcels.zip"

status = arcgis.features.FeatureLayerCollection.fromitem(item).manager.overwrite(dataPath)‍‍‍‍‍‍‍‍‍

error log:

Traceback (most recent call last):
  File "C:\Automation\Python\Jobs-ISD\AGOLUpdate\Code\AGOLUpdate.py", line 117, in <module>
    overwriteStatus = overwriteAGOLfeature(dataName,data.get('itemID'),data.get('zipData'))
  File "C:\Automation\Python\Jobs-ISD\AGOLUpdate\Code\AGOLUpdate.py", line 87, in overwriteAGOLfeature
    status = arcgis.features.FeatureLayerCollection.fromitem(item).manager.overwrite(dataPath)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis-1.3.0-py3.5.egg\arcgis\features\managers.py", line 771, in overwrite
    if related_data_item.update(data=data_file):
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis-1.3.0-py3.5.egg\arcgis\gis\__init__.py", line 4994, in update
    large_thumbnail)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis-1.3.0-py3.5.egg\arcgis\_impl\portalpy.py", line 2144, in update_item
    resp = self.con.post(path, postdata, files)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis-1.3.0-py3.5.egg\arcgis\_impl\connection.py", line 1076, in post
    resp = opener.open(req)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 472, in open
    response = meth(req, response)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 510, in error
    return self._call_chain(*args)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 444, in _call_chain
    result = func(*args)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 504: Gateway Time-out‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Intermittent 504 gateway timeouts with hosted feature layers

WASHSECTOR — Wed, 26 Sep 2018 17:27:09 GMT

Hi, What was the solution for this problem ?? I have the same problem. My script was running normally and when i have more data it stops at all. It always giving the error of HTTP Error 500: Internal ServerError and sometimes Error 504.

Re: Intermittent 504 gateway timeouts with hosted feature layers

PhilLarkin1 — Wed, 12 Dec 2018 22:31:44 GMT

This is in fact associated with the bug. The severity is logged as low.

BUG-000115253: When using the overwrite method from arcgis.features.managers.FeatureLayerCollectionManager, ArcGIS API for Python returns the error "Exception: Job Failed.''

BUG-000115253: When using the overwrite method from arcgis.features..

Re: Intermittent 504 gateway timeouts with hosted feature layers

PhilLarkin1 — Mon, 17 Dec 2018 17:26:42 GMT

The analyst with my case suggested using arcpy.sharing instead of the overwrite function in arcgis python api.

Introduction to arcpy.sharing—Sharing module | ArcGIS Desktop

Re: Intermittent 504 gateway timeouts with hosted feature layers

MichaelVolz — Mon, 17 Dec 2018 17:31:10 GMT

The OP said that he was using http requests in conjunction with AGOL. Will this be an issue when the next update is applied to AGOL in February based on the TLS security update?

Re: Intermittent 504 gateway timeouts with hosted feature layers

JohnFannon — Tue, 18 Dec 2018 12:31:23 GMT

As discussed above, we are using http requests via Python. Therefore whether any changes are required for the February TLS 1.2 update depends on the version of ArcGIS and therefore Python that is installed. In this case, our customer has ArcGIS 10.4.x, which installs Python 2.7.10, which uses TLS 1.2 by default, so should not require any changes.

Whether the python version uses TLS 1.2 can be determined by following the steps mentioned here - How to check if your Python app supports TLS 1.2 | Calazan.com.

However, if you're using ArcGIS Desktop to connect to ArcGIS Online or other connection methods, then you may need to take some action. There is more information available here - Esri Support Important Updates for the ArcGIS Platform and Transport Layer Security (TLS) Protocol Support.

Re: Intermittent 504 gateway timeouts with hosted feature layers

DavidRunneals2 — Tue, 08 Oct 2019 21:28:02 GMT

It's a bug that continues to persist and esri's workaround is basically to keep trying until it works... We've been getting 500, 502, 503, and 504 errors. I was told that the 500 and 504 errors are rolled into the 502 bug. Our process tries 3 times, and even then it still fails. Hopefully esri will realize that retrying multiple times is not a valid "workaround". I've also ran into these errors even through their ui when viewing data on a map and looking at the data table. After talking with some guys at the UC last year, it sounded like it might be a symptom of their back-end scaling up and caching an error response while it is. Kelly Gerrow‌

"Re-submit the requests to resolve the intermittent errors. Scripts or tools can be set up with retry on failure logic."

BUG-000124290 Making 'query' and 'applyEdits' requests to the ArcGIS Online hosted feature service intermittently return the 502 error.

BUG-000123780 Making 'query' and 'applyEdits' requests to the ArcGIS Online hosted feature service intermittently returns 503 errors.