Intermittent 504 gateway timeouts with hosted feature layers

[JohnFannon]

Hi

We have a customer for whom we have published some data as an ArcGIS online hosted feature service. We have a process that applies changes to the hosted layers based on changes in their database, including adds, updates and deletes. The process is written in python and the code is sending http requests to the layer's REST endpoint (i.e. addFeatures, updateFeatures, deleteFeatures).

The code is working fine generally and has been for some months, but we are increasingly seeing intermittent errors in the log for our process that are related to http requests, especially Error 504 (Gateway Timeout). I've looked into error 504 and this suggests a timeout is occuring between internal servers in ArcGIS Online. The error seems to occur most frequently when updating a specific layer (using updateFeatures), which has approx. 4500 features, but occasionally occurs with update requests to other layers with fewer features (approx. 500). It is totally intermittent, so during one run one request might fail, but then the next will succeed.

We have put in place some work arounds, such as making a number of attempts at the request when it fails and limiting the number of features updated per request. However, we are still seeing some instances of error 504.

I'm just wondering if anyone else has experience similar when applying regular updates to an AGOL hosted feature service?

Regards

John

GIS4BUSINESS

[PaulDavidson1]

Hi Kelly:

Yes, I can confirm that we're reading and writing to (2) hosted feature services in AGOL that are part of a Hosted Feature Layer.  They're part of a Relationship Class of a 1:many relationship. 

This is basically a Hydrant Inspection program.

We are using a Python script.

My solution is to loop the script 7 times.

With each loop, read an inspection table, determine an inspection status for new inspections since last loop, write status to the inspection table (the many) and to the Feature Class hosted layer record for that asset (the 1 of 1:many).  This can write a multiple number of records at once.

Then sleep for two minutes while more inspections occur.

After 7 loops, it dies out.

Task Scheduler is used to kick it off every 15 minutes so if it does crash, we're only down with our quasi-real time map for ~15 mins max.

This process is working well but of course feels like a hack but it works so I'm good with it.

Looks to me like we're using edit_features rather than append:

# here is where we update the two tables
    if len(InspectionEdits) > 0:
        EditInspectionResult = FeatureTable.edit_features(updates=InspectionEdits)
        EditHydrantResult = FeatureLayer.edit_features(updates=HydrantEdits)

I'll read up on using append.

Not exactly intuitive to do an append for an edit but if that's BP, we'll go that route.

I have found that we sometimes crash just reading the Feature Layer info, it's pretty random.

It's the web is how I tend to look at it.

Thanks for the append pointer.  I'll report back after a chance to work it over.

Could be a while since we're cranking along (it ain't broke...)

[PaulDavidson1]

Hey Kelly:

Looks like the postings are really just about using the append tool with the hosted layers.  Our issue is crashes in code which I'm pretty sure is due to timeouts.  Like Chuck says below, I haven't found anyway to set those values.

As Paul Barker lays out in his writings, we're basically doing upserts.  I think most updates in a database work as upserts. 

So I would think append is not applicable here but I'll dig into the ArcGIS python library when I can.

[JohnFannon]

Our issues didn't result from using append. We are sending requests directly to the REST endpoint via python (as the code was developed prior to the ArcGIS Python API) and getting the 504 gateway timeout errors. Any methods you are using via the ArcGIS Python API will ultimately result in a http request to the REST endpoint of the service using applyEdits, updateFeatures, addFeatures, or deleteFeatures operations, which are exactly the same operations we are using directly via python.

Also, setting any timout options on the client-side requests makes no difference, since these cover the client-side timeouts only and the error (504 gateway timeout) comes from an internal timeout on the server-side.

Within our code, we are capturing the http error and re-sending the same request again multiple times. In most cases this works, but occassionally we still get failures after multiple attempts. These don't result in crashes as we just log the error and continue.

[ChuckBenton]

I'm seeing similar issues with timeouts when appending or deleting content from a feature layer that's hosted on ArcGIS Online.  While not 100% consistent, it only happens with large files.  The feature layer I'm working with is quite large, over 40M parcels, so the work being performed takes a while, which is to be expected and acceptable.  I've looked for a timeout value that I can set to accommodate this without luck. Any suggestions are most welcome!

[PhilLarkin1]

I am also experiencing this behavior. After 4 months of flawless performance, an overwrite function has started failing intermittently due to a timeout. I'm using the python API (1.4.1) as show below. These are the hosted feature datasets I am updating:
http://spokanecounty.maps.arcgis.com/home/item.html?id=5805ca391e5f4c1ca6f0e58aee0d46a6 
http://spokanecounty.maps.arcgis.com/home/item.html?id=b6b1e7f8e12c4a9ba78af72076aee7af 
http://spokanecounty.maps.arcgis.com/home/item.html?id=848144f660a741f4b274360d494f15a1 

I believe BUG-000115253 may be associated. This was identified in 1.4.2. 

Bug synopsis: When using the overwrite method from arcgis.features.managers.FeatureLayerCollectionManager, ArcGIS API for Python returns the error "Exception: Job Failed.''

example:

import arcgis

gis = arcgis.gis.GIS("https://spokanecounty.maps.arcgis.com", <usr>, <pw>)

item = gis.content.get(itemID)

dataPath = "\\\\<path>\\Parcels.zip"

status = arcgis.features.FeatureLayerCollection.fromitem(item).manager.overwrite(dataPath)‍‍‍‍‍‍‍‍‍

error log:

Traceback (most recent call last):
  File "C:\Automation\Python\Jobs-ISD\AGOLUpdate\Code\AGOLUpdate.py", line 117, in <module>
    overwriteStatus = overwriteAGOLfeature(dataName,data.get('itemID'),data.get('zipData'))
  File "C:\Automation\Python\Jobs-ISD\AGOLUpdate\Code\AGOLUpdate.py", line 87, in overwriteAGOLfeature
    status = arcgis.features.FeatureLayerCollection.fromitem(item).manager.overwrite(dataPath)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis-1.3.0-py3.5.egg\arcgis\features\managers.py", line 771, in overwrite
    if related_data_item.update(data=data_file):
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis-1.3.0-py3.5.egg\arcgis\gis\__init__.py", line 4994, in update
    large_thumbnail)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis-1.3.0-py3.5.egg\arcgis\_impl\portalpy.py", line 2144, in update_item
    resp = self.con.post(path, postdata, files)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis-1.3.0-py3.5.egg\arcgis\_impl\connection.py", line 1076, in post
    resp = opener.open(req)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 472, in open
    response = meth(req, response)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 510, in error
    return self._call_chain(*args)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 444, in _call_chain
    result = func(*args)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\urllib\request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 504: Gateway Time-out‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[PhilLarkin1]

This is in fact associated with the bug. The severity is logged as low.

BUG-000115253: When using the overwrite method from arcgis.features.managers.FeatureLayerCollectionManager, ArcGIS API for Python returns the error "Exception: Job Failed.''

BUG-000115253: When using the overwrite method from arcgis.features.. 

[PhilLarkin1]

The analyst with my case suggested using arcpy.sharing instead of the overwrite function in arcgis python api. 

Introduction to arcpy.sharing—Sharing module | ArcGIS Desktop 

[MichaelVolz]

The OP said that he was using http requests in conjunction with AGOL.  Will this be an issue when the next update is applied to AGOL in February based on the TLS security update?

[JohnFannon]

As discussed above, we are using http requests via Python. Therefore whether any changes are required for the February TLS 1.2 update depends on the version of ArcGIS and therefore Python that is installed. In this case, our customer has ArcGIS 10.4.x, which installs Python 2.7.10, which uses TLS 1.2 by default, so should not require any changes.

Whether the python version uses TLS 1.2 can be determined by following the steps mentioned here - How to check if your Python app supports TLS 1.2 | Calazan.com.

However, if you're using ArcGIS Desktop to connect to ArcGIS Online or other connection methods, then you may need to take some action. There is more information available here - Esri Support Important Updates for the ArcGIS Platform and Transport Layer Security (TLS) Protocol S....

[DavidRunneals2]

It's a bug that continues to persist and esri's workaround is basically to keep trying until it works... We've been getting 500, 502, 503, and 504 errors. I was told that the 500 and 504 errors are rolled into the 502 bug. Our process tries 3 times, and even then it still fails. Hopefully esri will realize that retrying multiple times is not a valid "workaround". I've also ran into these errors even through their ui when viewing data on a map and looking at the data table. After talking with some guys at the UC last year, it sounded like it might be a symptom of their back-end scaling up and caching an error response while it is. Kelly Gerrow‌

"Re-submit the requests to resolve the intermittent errors. Scripts or tools can be set up with retry on failure logic."

BUG-000124290 Making 'query' and 'applyEdits' requests to the ArcGIS Online hosted feature service i...

BUG-000123780 Making 'query' and 'applyEdits' requests to the ArcGIS Online hosted feature service i...