Faster way to retrieve data from REST endpoint

Zeke — Sat, 11 Dec 2021 10:11:28 GMT

I have a script that goes out to REST endpoints to retrieve the various services. However, some of these services have hundreds of thousands of records. The max record limit on the endpoints is 1000, meaning I have to first get all the ObjectIDs and query the service in batches of 1000 based on the ObjectIDs. For 100,000 records, this means 100 queries. This is very time consuming. Is there a better way to get this data? Thanks.

for id in idstofetch:
                    # starting and ending object id for each group
                    idmin = str(id[0])
                    idmax = str(id[-1])

                    query_dict = {"token" : token["token"],
                                  "where" : "objectId >=" + idmin + "and objectid <=" + idmax,
                                  "outfields" : "*",
                                  "outSR" : 4326,
                                  "returnGeometry" : "true",
                                  "f" : "json"}

                    r = requests.get(url, query_dict, stream=True)

                    # build json file name
                    f = v.replace(' ', '_')
                    f = f.replace('(', '_')
                    f = f.replace(')', '_')
                    filenm = os.path.join(interim_folder, '{}_'.format(f) + dt + '_' + idmax) + '.json'

                    # create json file
                    with open(filenm, 'wb') as fd:
                        for chunk in r.iter_content(chunk_size=256):
                            fd.write(chunk)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Faster way to retrieve data from REST endpoint

RandyBurton — Mon, 26 Feb 2018 17:39:20 GMT

Perhaps export hosted feature layer using python would be of interest.

topic Re: Faster way to retrieve data from REST endpoint in Python Questions

Faster way to retrieve data from REST endpoint

Re: Faster way to retrieve data from REST endpoint