Faster way to retrieve data from REST endpoint

[Zeke]

I have a script that goes out to REST endpoints to retrieve the various services. However, some of these services have hundreds of thousands of records. The max record limit on the endpoints is 1000, meaning I have to first get all the ObjectIDs and query the service in batches of 1000 based on the ObjectIDs. For 100,000 records, this means 100 queries. This is very time consuming. Is there a better way to get this data? Thanks.

for id in idstofetch:
                    # starting and ending object id for each group
                    idmin = str(id[0])
                    idmax = str(id[-1])

                    query_dict = {"token" : token["token"],
                                  "where" : "objectId >=" + idmin + "and objectid <=" + idmax,
                                  "outfields" : "*",
                                  "outSR" : 4326,
                                  "returnGeometry" : "true",
                                  "f" : "json"}

                    r = requests.get(url, query_dict, stream=True)

                    # build json file name
                    f = v.replace(' ', '_')
                    f = f.replace('(', '_')
                    f = f.replace(')', '_')
                    filenm = os.path.join(interim_folder, '{}_'.format(f) + dt + '_' + idmax) + '.json'

                    # create json file
                    with open(filenm, 'wb') as fd:
                        for chunk in r.iter_content(chunk_size=256):
                            fd.write(chunk)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[RandyBurton]

Perhaps export hosted feature layer using python would be of interest.