topic Re: Using API doesn't return all data in ArcGIS API for Python Questions

Using API doesn't return all data

jaskainth — Thu, 27 Mar 2025 15:32:55 GMT

Hi, I'm trying to use the API in python to get all wildfires from this source - https://data-nifc.opendata.arcgis.com/datasets/nifc::wfigs-interagency-fire-perimeters/about (they use esri for their API). In the details it shows there are 27,977 rows. On the right side, after scrolling down on their page slightly, it shows

View API Resources

and I'm using the GeoJSON URL.

When I use this in python, doing something like this,

import geopandas as gpd import requests import os url = "https://services3.arcgis.com/T4QMspbfLg3qTGWY/arcgis/rest/services/WFIGS_Interagency_Perimeters/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json" cur_path = os.path.abspath(os.getcwd()) local_filepath = os.path.join(cur_path, 'wildfires.shp') resp = requests.get(url, stream=True) resp.raise_for_status() chunks_iter = resp.iter_content(chunk_size=1024 * 4) with open(local_filepath, 'wb') as file: for chunk in chunks_iter: file.write(chunk) df = gpd.read_file("wildfires.shp") df.shape # Returns (2000, 121)

I'm only getting 2000 rows returned when there should be ~28,000 rows. Does anyone know what the issue might be here?

Thanks!

Re: Using API doesn't return all data

JoshuaBixby — Thu, 27 Mar 2025 17:04:14 GMT

Look at the layer's property page, it states "Max Record Count: 2000", which is typical. You will need to make multiple calls to retrieve all of the data. You should read the REST API documentation about how to use the Result Offset and Result Record Count parameters.

Re: Using API doesn't return all data

Clubdebambos — Fri, 28 Mar 2025 07:34:37 GMT

Hi @jaskainth

Probably a few other options here too, see with ArcPy below. No multiple calls and will export all records. If exporting to a file geodatabase it would also hold domains and export attachments if the service had any.

import arcpy ## the filepath for the output shapefile out_shp = r'C:\Path\to\folder\wildfires.shp' ## the url to the feature layer url = 'https://services3.arcgis.com/T4QMspbfLg3qTGWY/arcgis/rest/services/WFIGS_Interagency_Perimeters/FeatureServer/0' arcpy.conversion.ExportFeatures(url, out_shp)

Here's a blog post with other options using the ArcGIS API for Python / ArcPy (focuses on file geodatabases but you can switch out for a shapefile in a folder). Exporting to shapefiles can sometimes cause issues with field truncation and some other limitations associated with a shapefile.

While the above with ArcPy works nicely, converting a pandas spatially enabled dataframe will fail as per the below. It is not truncating the field names.

from arcgis.features import FeatureLayer ## the filepath for the output shapefile shp_path = r"C:\Path\to\shapefile\folder\wildfires.shp" ## the url to the feature layer url = 'https://services3.arcgis.com/T4QMspbfLg3qTGWY/arcgis/rest/services/WFIGS_Interagency_Perimeters/FeatureServer/0' ## access feature layer of choice, here we are accessing the first layer fl = FeatureLayer(url) ## use query to return a FeatureSet object feature_set = fl.query() print(feature_set.sdf.shape) ## attempt to create a shapefile feature_set.sdf.spatial.to_featureclass(location=shp_path)

The shape is correct

(27997, 121)

Errors

cannot add field: 'poly_featu2' cannot add field: 'poly_sourc2' cannot add field: 'attr_ff_re2' cannot add field: 'attr_fire_9' cannot add field: 'attr_ics202' cannot add field: 'attr_incid6' cannot add field: 'attr_poo_j2' cannot add field: 'attr_poo_l6' ValueError: tuple.index(x): x not in tuple

You can probably manipulate the dataframe to overcome these.