Using API doesn't return all data

[jaskainth]

Hi, I'm trying to use the API in python to get all wildfires from this source - https://data-nifc.opendata.arcgis.com/datasets/nifc::wfigs-interagency-fire-perimeters/about (they use esri for their API). In the details it shows there are 27,977 rows. On the right side, after scrolling down on their page slightly, it shows

View API Resources

and I'm using the GeoJSON URL. 

When I use this in python, doing something like this,

import geopandas as gpd
import requests
import os
url = "https://services3.arcgis.com/T4QMspbfLg3qTGWY/arcgis/rest/services/WFIGS_Interagency_Perimeters/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json"
cur_path = os.path.abspath(os.getcwd())
local_filepath = os.path.join(cur_path, 'wildfires.shp')
resp = requests.get(url, stream=True)
resp.raise_for_status()
chunks_iter = resp.iter_content(chunk_size=1024 * 4)
with open(local_filepath, 'wb') as file:
    for chunk in chunks_iter:
        file.write(chunk)
df = gpd.read_file("wildfires.shp")
df.shape # Returns (2000, 121)

I'm only getting 2000 rows returned when there should be ~28,000 rows. Does anyone know what the issue might be here?

Thanks!

[JoshuaBixby · Accepted solution]

Look at the layer's property page, it states "Max Record Count: 2000", which is typical.  You will need to make multiple calls to retrieve all of the data.  You should read the REST API documentation about how to use the Result Offset and Result Record Count parameters.

[Clubdebambos]

Hi @jaskainth 

Probably a few other options here too, see with ArcPy below. No multiple calls and will export all records. If exporting to a file geodatabase it would also hold domains and export attachments if the service had any.

import arcpy

## the filepath for the output shapefile
out_shp = r'C:\Path\to\folder\wildfires.shp'

## the url to the feature layer
url = 'https://services3.arcgis.com/T4QMspbfLg3qTGWY/arcgis/rest/services/WFIGS_Interagency_Perimeters/FeatureServer/0'

arcpy.conversion.ExportFeatures(url, out_shp)

Here's a blog post with other options using the ArcGIS API for Python / ArcPy (focuses on file geodatabases but you can switch out for a shapefile in a folder). Exporting to shapefiles can sometimes cause issues with field truncation and some other limitations associated with a shapefile.

While the above with ArcPy works nicely, converting a pandas spatially enabled dataframe will fail as per the below. It is not truncating the field names.

from arcgis.features import FeatureLayer

## the filepath for the output shapefile
shp_path = r"C:\Path\to\shapefile\folder\wildfires.shp"

## the url to the feature layer
url = 'https://services3.arcgis.com/T4QMspbfLg3qTGWY/arcgis/rest/services/WFIGS_Interagency_Perimeters/FeatureServer/0'

## access feature layer of choice, here we are accessing the first layer
fl = FeatureLayer(url)

## use query to return a FeatureSet object
feature_set = fl.query()

print(feature_set.sdf.shape)

## attempt to create a shapefile
feature_set.sdf.spatial.to_featureclass(location=shp_path)

The shape is correct

(27997, 121)

Errors

cannot add field: 'poly_featu2'
cannot add field: 'poly_sourc2'
cannot add field: 'attr_ff_re2'
cannot add field: 'attr_fire_9'
cannot add field: 'attr_ics202'
cannot add field: 'attr_incid6'
cannot add field: 'attr_poo_j2'
cannot add field: 'attr_poo_l6'

ValueError: tuple.index(x): x not in tuple

You can probably manipulate the dataframe to overcome these.