504 Gateway Time-Out when uploading raster with copy_raster

IceyeAnalytics · ‎01-24-2023

I have had a new problem come up with previously working code. The background is that we have a cronjob running on an ec2 to upload 4 tifs twice daily. Below is how I call copy_raster to upload the tifs.

upload_layer = copy_raster(input_raster=tif,
    output_name=upload_name,
    folder=agol_folder,
    gis=gis)

For the last two days, this has been giving a 504 Gateway Time-out error (below) after working previously for weeks. A few other notes: the 4 tifs are around 6Mb each and when I turn the displayProgress flag to true it shows that each uploads in around 3 seconds but then the timeout seems to happen on the processing step since the next upload progress bar does not appear. I have checked the tifs - they are not corrupt and visualize properly in a local GIS platform. I am experiencing this same error both on the remote ec2 machine and my local one. Any help is appreciated, thank you.

Traceback (most recent call last):
File "/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/python3.9/site-packages/arcgis/gis/_impl/_con/_connection.py", line 993, in _handle_response
data = resp.json()
File "/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "file.py", line 471, in <module>
updateAGOL(
File "file.py", line 410, in updateAGOL
upload_tifs(gis, tif_dir, agol_folder, date_str_id)
File "file.py", line 74, in upload_tifs
upload_layer = copy_raster(input_raster=tif,
File "/python3.9/site-packages/arcgis/raster/analytics.py", line 1651, in copy_raster
return gis._tools.rasteranalysis.copy_raster(
File "/python3.9/site-packages/arcgis/_impl/tools.py", line 13022, in copy_raster
gpjob = self._tbx.copy_raster(
File "<string>", line 1245, in copy_raster
File "/python3.9/site-packages/arcgis/geoprocessing/_support.py", line 468, in _execute_gp_tool
job_info = gptool._con.post(submit_url, gp_params)
File "/python3.9/site-packages/arcgis/gis/_impl/_con/_connection.py", line 1528, in post
return self._handle_response(
File "/python3.9/site-packages/arcgis/gis/_impl/_con/_connection.py", line 996, in _handle_response
raise Exception(resp.text)
Exception: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>Microsoft-Azure-Application-Gateway/v2</center>
</body>
</html>

RhettZufelt · ‎01-24-2023

I generally chalk that up to the extreme flakiness of hosted services on AGOL (at least with services3.arcgis.com). Generally, you can try again later and "usually" works.

Don't bother looking at the ArcGIS Online Health page, as it doesn't pick up on this kind of thing, it will show everything green, even when not.

I often load a web map using AGOL data and look at the 'An error occured loading this layer' errors, or, some layers will be "loading" the entire time, but never finish. Then reload the map until you no longer get the error and all the layers load (it will be random FC's that fail to load, then on releoad, often a different set that throws the error). (Just took 6 reloads over 10 minutes to get the data to load again, now my python can access it also).

Since the esri servers can never be trusted to "stay up" long enough for processing larger datasets, I have modified my scripts to keep track of what has already been processed (add to list of processed ID's), then I embed another try: block within a while True:.

This way, if it gets the 504 error (you can count on it at some time), it will continue to process the items that are not in my processed list.

Often, it will run to completion the first try, but often, it has to keep trying again and again. (just did 16 retries and is going again(though, this time it threw 502, 503, and 504 errors).)

Not sure if this is what is going on with yours, but I do see this frequently with nothing wrong with the data, and the same code will work later on when the services are all back up.

R_

IceyeAnalytics · ‎01-26-2023

Thank you, that seems to be what was going on. For anyone coming across this post with the same issue we found a way to get around this, at least for the moment. Signing in to the GIS instance each time before uploading a tif seems to work much more consistently.