2000 rows limitation with api export json data

Question

Hi all. My name is Leah, I am working as a data specialist to help my team with reporting and data analysis. We have a GIS partner who built a web app for us. The web app is used for collecting data from offline environments, and it works great with our workflow. We also have a powerBI dashboard to use the data collected from this web app for reporting. I am able to set up the Rest API with auth token to refresh the data from a feature layer, however, I can export 4200 rows from the ArcGIS online (https:///arcgis/rest/services//FeatureServer/0). Only 2000 rows were exported if I use Rest API (https:///arcgis/rest/services//FeatureServer/0/query?where=1%3D1&outFields=*&returnGeometry=false&resultRecordCount=0&f=json&token=token). I followed another post https://support.esri.com/en/technical-article/000012383 to change the maxRecordCount to 8000. Even so, I still get only 2000 rows from my export json file. Does anyone have an idea why this would be happening? Is anything I can do to export the entire dataset from a feature layer host on ArcGIS online? If not, is there any workaround for PowreBI data refreshing? Thanks for the help!

ArthurSmith · Answer

Came across this limitation myself and found this thread.  Used @Raul_Jimenez 's reccomendation and used the resultOffset parameter in an ArcPy script.  Here is the code if anyone else needs to do something similar.

def Map_Service_Batch_Download(fs_url, batch_size, download_folder, out_fc):
# Get count of records in feature service and # of download batches needed
record_cnt = int(str(arcpy.management.GetCount(fs_url)))
downloads_necessary = math.ceil(record_cnt/batch_size)

# Download .geojson for each batch based on total # of records, 2000 at a time
for dl_cnt in list(range(downloads_necessary)):
start_record = dl_cnt*batch_size
urllib.request.urlretrieve(fs_url + "/query?outFields=*&where=1%3D1&f=geojson&resultOffset=" + str(start_record) + "&resultRecordCount=" + str(batch_size), dl_dir + "Weld_Parcels_DL_" + str(dl_cnt) + ".geojson")

# Loop through all the downloaded .geojsons and load to output feature class
geojson_cnt = 0
for geojson in os.listdir(download_folder):
arcpy.JSONToFeatures_conversion(download_folder + geojson, "memory/parcels_temp", "POLYGON")
if geojson_cnt == 0:
arcpy.arcpy.management.CopyFeatures("memory/parcels_temp", out_fc)
else:
arcpy.management.Append("memory/parcels_temp", out_fc)
os.remove(download_folder + geojson)
geojson_cnt += 1

Map_Service_Batch_Download(weld_parcel_url, 2000, dl_dir, parcels_from_geojson)

AW_Grove · Answer

Hi @Raul_Jimenez - do you have an example query where you have used the resultOffset option? Below is what I am using right now, but seeing max of 2000 records. I don't see the resultOffset parameter in the query documentation so I am wondering if it is supported with this method.

(https://services3.arcgis.com/xyz/arcgis/rest/services/xyz/FeatureServer/13/query? where=%3D&outFields=*&returnGeometry=false&resultRecordCount=0&f=json) I am bringing the data into PowerBI if it matters.

Raul_Jimenez · Answer

I have never been able to increase the maxRecordCount above 2000, but you have two options:

1) Use the "resultOffset" parameter (which is the pagination mechanism): <- this is the one I would recommend

2) Use the asynchronous/job "https://[root]/content/users/[userName]/export" endpoint to generate a file.

Another page that you might want to check is the Feature service page within the "Mapping APIs and locations services guide" (it is a conceptual guide about many of the cross concepts of the ArcGIS system, including the REST APIs).

I hope this helps.