Intermittent 504 gateway timeouts with hosted feature layers

JohnFannon · ‎12-07-2017

Hi

We have a customer for whom we have published some data as an ArcGIS online hosted feature service. We have a process that applies changes to the hosted layers based on changes in their database, including adds, updates and deletes. The process is written in python and the code is sending http requests to the layer's REST endpoint (i.e. addFeatures, updateFeatures, deleteFeatures).

The code is working fine generally and has been for some months, but we are increasingly seeing intermittent errors in the log for our process that are related to http requests, especially Error 504 (Gateway Timeout). I've looked into error 504 and this suggests a timeout is occuring between internal servers in ArcGIS Online. The error seems to occur most frequently when updating a specific layer (using updateFeatures), which has approx. 4500 features, but occasionally occurs with update requests to other layers with fewer features (approx. 500). It is totally intermittent, so during one run one request might fail, but then the next will succeed.

We have put in place some work arounds, such as making a number of attempts at the request when it fails and limiting the number of features updated per request. However, we are still seeing some instances of error 504.

I'm just wondering if anyone else has experience similar when applying regular updates to an AGOL hosted feature service?

Regards

John

GIS4BUSINESS

JamesCrandall · ‎07-02-2020

This issue has gone from intermittent (maybe 1 time each 2 weeks) to about 10 times per 24 hours on a python script run that simply queries the REST interface of a hosted feature service. We get 400 errors when simply selecting the last datetime stamp value:

qFlParams = urllib.urlencode({'f': 'json',
                              'outFields': 'location_timestamp',
                              'orderByFields': 'location_timestamp DESC',
                              'resultRecordCount' : '1',
                              'where': "created_user = '{}'".format(inName),
                              'token': token
                             })

reqFl = urllib2.Request(urlTrackerHistory + '/query', qFlParams)
response = urllib2.urlopen(reqFl)
gFeats = json.load(response)
#print gFeats‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

gFeats respone comes back as:

{u'error': {u'message': u'Cannot perform query. Invalid query parameters.', u'code': 400, u'details': [u'Unable to perform query. Please check your parameters.']}}‍

But one subsequent attempt, it succeeds.

kmsmikrud · ‎05-13-2021

Hello,

Thanks for submitting the BUGS regarding the 50x errors, I am getting SO very many of these with the post-survey script I am trying to run to the point its beyond frustrating. The python script is using the update cursor to populate records of data collected in the Field Maps app. When the script successfully runs it will run for an hour. At the moment between yesterday and today I have not even been able to make it thru 275 records in the hosted feature layer to update without getting these timeout errors to the point I've gotten 5 in the last hour. I am at the office and have a good wired network connection. At the moment the script I am using does not have the logic to trigger it to retry the loops. I am a pretty novice programmer and am happy to have the script working as it should with the tasks/metrics needed that I am completely annoyed now to be getting server connection errors.

Some days even the AGOL hosted feature layer tables are data error after data error yet the AGOL health status is all good. Its hard to feel very confident in using the Esri cloud if these services are so intermittent in these script connections, or viewing in AGOL, and also there are the sync errors for apps like Field Maps that need to connect to these services to sync data or when downloading an offline map. Seriously its hard to have much confidence in the cloud.

In the last 5 minutes I've seen three 504 timeout errors as the scripts displays messages.

Very frustrating and in reviewing the BUG status even though they were submitted 2 years ago with High severity nothing seems to be done except for users to add logic/workarounds to compensate.

Kathy

andrei_ste · ‎05-28-2021

I have the same issue while updating and adding new features to a hosted feature service in ArcGIS Online, using Python API for ArcGIS. I have two scripts that run daily, on different periods of the day, update different tables, but use the same hosted feature services. One runs successfully almost everytime, the other encounters this error. In the first 2-3 months I didn't have this issue and now I need to run 3-4 times the script to run it successfully which isn't pleasant because I was using the operating system task manager to run the scripts automatically at a given hour. I assume the process takes now a little bit longer and exceeds the timeout settings of the service which I can not change. Sometimes I get an error that the token is invalid, probably the process took to long and the token expired. Did someone find a solution to it?

andrei_ste · ‎06-22-2021

I made a few modifications to the script that are not related to this problem and now it works again, without the above error. I am still using the edit_features method from the Python for ArcGIS API to add and update features. For now I do not know what was the cause of the problem. Hope it helps

PeterMilenkovic · ‎06-03-2021

Also experiencing issues, my script was working fine in a pro notebook, but now that I'm running it via a .py file using task scheduler it throws these errors.

One way I thought I could get around it is via looping the adds and getting the feature layer for each add so it reestablishes a connection... This ran and went to the except after a few loops, then it timed out..

NewSharpsInspections = pd.merge(left = SharpsBin_item_df, right = SDE_SharpsInspectionsQuery, how = 'inner', on ='Asset_ID')


    #looping through dataframe to add 200 inspections at a time, method doesnt like more than 200

    RecordCount = NewSharpsInspections.shape[0]

    try:

        for x in range(0,RecordCount,500):
            sv = x
            fv = x+500
            print('inserting rows ' + str(sv) + ' to ' + str(fv))
            NewSharpsInspections_FS = FeatureSet.from_dataframe(NewSharpsInspections.iloc[sv:fv].copy())
            SharpsInspections_item = gis.content.get('decb8a23f5af4abc8ffc295874d5e662')
            SharpsInspections_FL = SharpsInspections_item.layers[0]
            SharpsInspections_FL.edit_features(adds = NewSharpsInspections_FS)
            
    except:
        print('it timed out, let\'s try this...')
        SharpsInspections_item = gis.content.get('decb8a23f5af4abc8ffc295874d5e662')
        SharpsInspections_FL = SharpsInspections_item.layers[0]
        SharpsInspections_item_df =  SharpsInspections_FL.query().sdf
        InspectionsAdded = SharpsInspections_item_df.shape[0]
        SharpsInspections_Remaining = NewSharpsInspections.iloc[InspectionsAdded:]
     
        
        for x in range(InspectionsAdded,RecordCount,500):
            sv = x
            fv = x+500
            print('inserting rows ' + str(sv) + 'to ' + str(fv))
            NewSharpsInspections_FS = FeatureSet.from_dataframe(SharpsInspections_Remaining.iloc[sv:fv].copy())
            SharpsInspections_item = gis.content.get('decb8a23f5af4abc8ffc295874d5e662')
            SharpsInspections_FL = SharpsInspections_item.layers[0]
            SharpsInspections_FL.edit_features(adds = NewSharpsInspections_FS)

Might have to resort to doing an append instead of using edit features, but i have to convert my feature set to a feature collection...?

https://developers.arcgis.com/python/api-reference/arcgis.features.toc.html?highlight=append#arcgis....

philnagel · ‎06-23-2021

We are also experiencing this issue with some scripts. It seems that more complex geometries trigger the behavior here. Even adding one feature with such a complex geometry can cause this timeout error.

I noticed that one of the bugs mentioned above (BUG-000123780) has had an update made to it yesterday, and has the status "In Product Plan", which I believe means they are at least thinking they have a possible fix for it.

NathanHeickLACSD · ‎11-24-2021

Good Morning,

It's nice to see the problem I have been having for a long-time is an ArcGIS Online issue. I am getting a timeout error with a few thousand point features using the addFeatures endpoint. It seems like it happens 20% of the time.

Nathan

NathanHeickLACSD · ‎02-11-2022

After accepting this is a longstanding issue that could continue on for a lot longer, I just modified my script to start the entire sequence over again if one of these errors occurred. This meant wrapping a large block of REST calls into one try-except block. However, it has worked very well. I have it wait 30 minutes between attempts and I try 5 times. I would be very displeased if I needed 24-7 support.

VincentLantaca · ‎02-11-2022

I am getting Error 504 on feature layers hosted on AGOL as well. Based on some testing I've done it seems to be a problem with AGOL. I have created a small test that demonstrates the issue:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>


    <script src="https://unpkg.com/@esri/arcgis-rest-request@3.4.3/dist/umd/request.umd.min.js"></script>
    <script src="https://unpkg.com/@esri/arcgis-rest-feature-layer@3.4.3/dist/umd/feature-layer.umd.min.js"></script>
    <script src="https://unpkg.com/@esri/arcgis-rest-auth@3.4.3/dist/umd/auth.umd.min.js"></script>

    <script>
        function getTimeDelta(start, stop){
            var timeDelta = parseInt(stop - start);
            var days = parseInt(timeDelta / 86400000);
            var daysRem = timeDelta - (days * 86400000);
            var hours = parseInt(daysRem / 3600000);
            var hoursRem = daysRem - (hours * 3600000);
            var minutes = parseInt(hoursRem / 60000);
            var minutesRem = hoursRem - (minutes * 60000);
            var seconds = parseInt(minutesRem / 1000);
            var milliseconds = minutesRem - (seconds * 1000);

            var results = {
                days: days,
                hours: hours,
                minutes: minutes,
                seconds: seconds,
                milliseconds: milliseconds
            }

            return results;
        }

        const username = ""; // AGOL username
        const password = ""; // AGOL password
        const portalRestUrl = "https://domain.com/sharing/rest"; // AGOL portal url
        const hostedFeatureLayerUrl = "https://utility.arcgis.com/usrsvcs/servers/2d48cb458feb445394d02ddzb7gddb43/rest/services/FOLDER/SERVICE_NAME/FeatureServer/2"; // hosted feature layer url

        // create a new portal session
        const session = new arcgisRest.UserSession({
            username: username,
            password: password,
            portal: portalRestUrl
        });

        // list of OBJECTIDs that will be updated
        var objectidList = [
            12226684,
            12226688,
            12226685,
            12226686,
            12227085,
            12226687,
            12227082,
            12226690,
            12226689,
            12227882,
            12227485,
            12227482,
            12227486,
            12227484,
            12228282,
            12227084,
            12227083,
            12227483,
            12226682,
            12226683
        ];

        // get the current time
        var startTime = new Date();

        objectidList.forEach((oid) => {

            // perform an update on each feature from their OBJECTID
            arcgisRest.updateFeatures({
                url: hostedFeatureLayerUrl,
                features: [{"attributes":{"OBJECTID":oid,"FIELD_NAME":16171}}],
                authentication: session
            }).then(updateResults => {
                var currTime = new Date();
                console.log('time delta:', getTimeDelta(startTime, currTime));
                console.log('results of updateFeatures', updateResults);
                console.log(updateResults['updateResults'][0]['objectId']);
                console.log('success', updateResults['updateResults'][0]['success']);
            }, (reason) => {
                var currTime = new Date();
                console.log('time delta:', getTimeDelta(startTime, currTime));
                console.log('failed', reason);
            });
        });

    </script>

</head>
<body>
    
</body>
</html>

Using ArcGIS REST JS, the above sample takes a hosted feature layer and uses updateFeatures to update a single field for a set of records specified by a list of OBJECTIDs. For every call to updateFeatures, the time it takes for the request to complete is logged in the browser console. From my testing, the code consistently starts giving Error 504s for requests that take longer than 30 seconds. (Despite that the timeout for this service is set to 60 seconds on the server from ArcGIS Server Manager).

Individual updateFeatures requests may take longer on some layers compared to others. So for some layers you may have to use more or less records to get requests to start taking longer than the apparent 30 second timeout.

If I do the exact same thing except making requests directly to our ArcGIS Server, requests will succeed even if they take longer than 30 seconds to complete, which is what we expect:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>


    <script src="https://unpkg.com/@esri/arcgis-rest-request@3.4.3/dist/umd/request.umd.min.js"></script>
    <script src="https://unpkg.com/@esri/arcgis-rest-feature-layer@3.4.3/dist/umd/feature-layer.umd.min.js"></script>
    <script src="https://unpkg.com/@esri/arcgis-rest-auth@3.4.3/dist/umd/auth.umd.min.js"></script>

    <script>
        function getTimeDelta(start, stop){
            var timeDelta = parseInt(stop - start);
            var days = parseInt(timeDelta / 86400000);
            var daysRem = timeDelta - (days * 86400000);
            var hours = parseInt(daysRem / 3600000);
            var hoursRem = daysRem - (hours * 3600000);
            var minutes = parseInt(hoursRem / 60000);
            var minutesRem = hoursRem - (minutes * 60000);
            var seconds = parseInt(minutesRem / 1000);
            var milliseconds = minutesRem - (seconds * 1000);

            var results = {
                days: days,
                hours: hours,
                minutes: minutes,
                seconds: seconds,
                milliseconds: milliseconds
            }

            return results;
        }

        var featureServiceUrl = "https://domain.com/server/rest/services/FOLDER/SERVICE_NAME/FeatureServer/2/updateFeatures";
        var generateTokenUrl = "https://domain.com/server/tokens/generateToken";
        var username = "";
        var password = "";

        // call generateToken directly to a ArcGIS Server
        async function generateTokenFromServer(url = '', username = '', password = ''){
            var myHeaders = new Headers();
            myHeaders.append("Content-Type", "application/x-www-form-urlencoded");

            var urlencoded = new URLSearchParams();
            urlencoded.append("f", "json");
            urlencoded.append("username", username);
            urlencoded.append("password", password);
            urlencoded.append("client", "requestip");

            var requestOptions = {
                method: 'POST',
                headers: myHeaders,
                body: urlencoded,
                redirect: 'follow'
            };

            const response = await fetch(url, requestOptions);
            console.log(response);
            return response.json();
        }

        // get a token for calling services
        async function getToken(url, username, password){
            var generateTokenResponse = await generateTokenFromServer(url, username, password);
            return generateTokenResponse['token'];
        }

        // update a field 'FIELD_NAME' for record with provided objectid
        async function updateRecord(url, objectid, fieldValue, token){
            var myHeaders = new Headers();
            myHeaders.append("Content-Type", "application/x-www-form-urlencoded");

            var urlencoded = new URLSearchParams();
            urlencoded.append("f", "json");
            urlencoded.append("features", JSON.stringify([
                {
                    "attributes": {
                        "OBJECTID": objectid,
                        "FIELD_NAME": fieldValue
                    }
                }
            ]));
            urlencoded.append("token", token);

            var requestOptions = {
                method: 'POST',
                headers: myHeaders,
                body: urlencoded,
                redirect: 'follow'
            };

            const response = await fetch(url, requestOptions);
            console.log(response);
            return response.json();
        }


        // list of OBJECTIDs that will be updated
        var objectidList = [
            12226684,
            12226688,
            12226685,
            12226686,
            12227085,
            12226687,
            12227082,
            12226690,
            12226689,
            12227882,
            12227485,
            12227482,
            12227486,
            12227484,
            12228282,
            12227084,
            12227083,
            12227483,
            12226682,
            12226683
        ];

        getToken(generateTokenUrl, username, password).then(token => {

            // get the current time
            var startTime = new Date();
            var successCount = 0;

            // for each objectid in the list, update it and print how much time has elapsed
            objectidList.forEach(oid => {
                updateRecord(featureServiceUrl, oid, 16171, token).then(result => {
                    var currTime = new Date();
                    console.log('time delta:', getTimeDelta(startTime, currTime));
                    successCount += 1;
                    console.log('success count: ', successCount);
                    console.log('results of updateFeatures', result);
                    console.log(result['updateResults'][0]['objectId']);
                    console.log('success', result['updateResults'][0]['success']);
                }, reason => {
                    var currTime = new Date();
                    console.log('time delta:', getTimeDelta(startTime, currTime));
                    console.log('failed', reason);
                });
            });

        });
    </script>

</head>
<body>
    
</body>
</html>

So it seems like there is a 30 second timeout specific to AGOL hosted feature layers. Even if updateFeatures operations actually succeed and all the records are updated in the underlying feature class, AGOL will still give a 504 for the requests that take longer than 30 seconds, which is frustrating because you don't know whether or not the request actually succeeded.

In my examples I did multiple single-edits, but I would guess you would see the same timeout for multiple edits in a single request.

NathanHeickLACSD · ‎02-11-2022

In my case Vincent, I was getting the 504 errors and when I reviewed the data, the requests were failing somewhere in the middle. I even load the data in chunks. When everything is going well, my addFeatures request completes well under 30 seconds. When it's not working, it just times out on AGOL's side. I definitely like your idea of checking it over and over. Did you see that the issue happened at certain times of day?