topic Re: Get all data using REST GET request in ArcGIS REST APIs and Services Questions

Get all data using REST GET request

PerfectCamp — Mon, 07 Jun 2021 22:18:12 GMT

First time user. Please forgive my ignorance. I have read the instructions and did some searches on this discussion board. Still have some beginner questions...

I'm trying to get a GeoJSON output of all BLM land in the US. I am on this query page:

https://gis.blm.gov/arcgis/rest/services/lands/BLM_Natl_SMA_Cached_BLM_Only/MapServer/2/query

I'm guessing this the correct page of instructions on how to query the data. Can you confirm?

https://developers.arcgis.com/rest/services-reference/enterprise/query-map-service-layer-.htm

On the query form I am leaving all the defaults except for the following:

Format changed to GeoJSON

I pressed the GET button and see this response:

{"error":{"code":400,"message":"","details":[]}}

Here is the GET request:
https://gis.blm.gov/arcgis/rest/services/lands/BLM_Natl_SMA_Cached_BLM_Only/MapServer/2/query?where=&text=&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&distance=&units=esriSRUnit_Foot&relationParam=&outFields=&returnGeometry=true&returnTrueCurves=false&maxAllowableOffset=&geometryPrecision=&outSR=&havingClause=&returnIdsOnly=false&returnCountOnly=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false&gdbVersion=&historicMoment=&returnDistinctValues=false&resultOffset=&resultRecordCount=&returnExtentOnly=false&datumTransformation=&parameterValues=&rangeValues=&quantizationParameters=&featureEncoding=esriDefault&f=geojson

I'm assuming there are some required fields not filled out. I then tried to add the following

Outfields = *
Where = 1=1

The GET request spins for about 10 min then returns an error:
ArcGIS Web Adaptor. Could not access any server machines. Please contact your system administrator.

I am trying different parameters to get it to work and I have a hunch that I need to provide an Input Geometry. I'm confused exactly what this field does. If I want all the data, do I just specify an envelope rectangle around the entire globe i.e. x and y values of -180 to 180 and -90 to 90 and say within? I do see some examples in the docs but don't understand why I need to specify an input geometry.

Thanks for helping me understand this. Go easy on me 🙂 Peace!

Re: Get all data using REST GET request

jcarlson — Mon, 07 Jun 2021 22:17:55 GMT

Input geometry filters the result by some extent or custom spatial relationship, like intersecting with some point.

I'm able to get results from this when setting the returnGeometry parameter to False, if thta's helpful.

Re: Get all data using REST GET request

DavidPike — Mon, 07 Jun 2021 22:38:49 GMT

Have a look when the query is OBJECTID=1, the geometry for all of those features is probably just too much for the request. Possibly iterate through all the objectIDs as separate requests and collate?

Re: Get all data using REST GET request

BenRomlein — Mon, 07 Jun 2021 22:41:05 GMT

these are large geometries, so I'm guessing the server is choking trying to return everything all at once.

You can limit your return with a where clause, or an input geometry filter.

Or if you really want everything, try using a combination of ResultRecordCount and ResultOffset to get the data a little bit at a time.

https://developers.arcgis.com/rest/services-reference/enterprise/query-feature-service-layer-.htm

Re: Get all data using REST GET request

PerfectCamp — Mon, 07 Jun 2021 22:59:26 GMT

Thanks so much to those who responded so fast!! You guys are great! Yes I was able to get a list of all the OBJECTID's then I'm writing a script to grab just one at a time. I did not have to use Input Geometry, just had to limit the response to something manageable by specifying WHERE OBJECTID = 1 and it seems to work without choking! Awesome! I'll get on it.

Re: Get all data using REST GET request

JoshuaBixby — Mon, 07 Jun 2021 23:48:32 GMT

If you are wanting/needing the "raw" data to do analysis locally, I strongly discourage obtaining the data set from a GIS service. GIS services are handy for quite a few things, but they make terrible data exchanges for dense data sets, at least in my opinion. Additionally and specifically for this data set, it is being served up in Web Mercator, which is a garbage projection unless you making/using basic browser-based maps.

If you visit BLM National Surface Management Agency Area Polygons - National Geospatial Data Asset (NGDA) - data.doi.gov, you can download the data, which looks to be an 850 MB zip file. Good luck downloading an 850 MB data set via map service, that will be fun and slow.