2000 rows limitation with api export json data

[LeahYan]

Hi all. My name is Leah, I am working as a data specialist to help my team with reporting and data analysis. We have a GIS partner who built a web app for us. The web app is used for collecting data from offline environments, and it works great with our workflow. We also have a powerBI dashboard to use the data collected from this web app for reporting. 

I am able to set up the Rest API with auth token to refresh the data from a feature layer, however, I can export 4200 rows from the ArcGIS online (https://<services-url>/arcgis/rest/services/<layer-name>/FeatureServer/0).

Only 2000 rows were exported if I use Rest API  (https://<services-url>/arcgis/rest/services/<layer-name>/FeatureServer/0/query?where=1%3D1&outFields=*&returnGeometry=false&resultRecordCount=0&f=json&token=token).

I followed another post https://support.esri.com/en/technical-article/000012383 to change the maxRecordCount to 8000. Even so, I still get only 2000 rows from my export json file. 

Does anyone have an idea why this would be happening? Is anything I can do to export the entire dataset from a feature layer host on ArcGIS online? If not, is there any workaround for PowreBI data refreshing?

Thanks for the help!

[JoshuaBixby]

Have you read the Query (Feature Service/Layer)—ArcGIS REST APIs | ArcGIS Developers documentation?  Specifically, the parts about resultType and max record count?

[Raul_Jimenez]

I have never been able to increase the maxRecordCount above 2000, but you have two options:

1) Use the "resultOffset" parameter (which is the pagination mechanism): <- this is the one I would recommend

2) Use the asynchronous/job "https://[root]/content/users/[userName]/export" endpoint to generate a file.

Another page that you might want to check is the Feature service page within the "Mapping APIs and locations services guide" (it is a conceptual guide about many of the cross concepts of the ArcGIS system, including the REST APIs).

I hope this helps.

[AW_Grove]

Hi @Raul_Jimenez - do you have an example query where you have used the resultOffset option? Below is what I am using right now, but seeing max of 2000 records. I don't see the resultOffset parameter in the query documentation so I am wondering if it is supported with this method. 

(https://services3.arcgis.com/XXXXXXXXXXX/arcgis/rest/services/XXXXXXXXX/FeatureServer/13/query?

where=%3D&outFields=*&returnGeometry=false&resultRecordCount=0&f=json)

I am bringing the data into PowerBI if it matters.

[ArthurSmith]

Came across this limitation myself and found this thread.  Used @Raul_Jimenez 's reccomendation and used the resultOffset parameter in an ArcPy script.  Here is the code if anyone else needs to do something similar.

def Map_Service_Batch_Download(fs_url, batch_size, download_folder, out_fc):
# Get count of records in feature service and # of download batches needed
record_cnt = int(str(arcpy.management.GetCount(fs_url)))
downloads_necessary = math.ceil(record_cnt/batch_size)

# Download .geojson for each batch based on total # of records, 2000 at a time
for dl_cnt in list(range(downloads_necessary)):
start_record = dl_cnt*batch_size
urllib.request.urlretrieve(fs_url + "/query?outFields=*&where=1%3D1&f=geojson&resultOffset=" + str(start_record) + "&resultRecordCount=" + str(batch_size), dl_dir + "Weld_Parcels_DL_" + str(dl_cnt) + ".geojson")

# Loop through all the downloaded .geojsons and load to output feature class
geojson_cnt = 0
for geojson in os.listdir(download_folder):
arcpy.JSONToFeatures_conversion(download_folder + geojson, "memory/parcels_temp", "POLYGON")
if geojson_cnt == 0:
arcpy.arcpy.management.CopyFeatures("memory/parcels_temp", out_fc)
else:
arcpy.management.Append("memory/parcels_temp", out_fc)
os.remove(download_folder + geojson)
geojson_cnt += 1

Map_Service_Batch_Download(weld_parcel_url, 2000, dl_dir, parcels_from_geojson)