topic Re: ArcGIS Server REST API Query in ArcGIS Enterprise Questions

ArcGIS Server REST API Query

mpboyle — Wed, 14 Jan 2015 15:09:30 GMT

I'm wondering if it's possible to create a script that loops through an ArcGIS Server REST API query in order to retrieve all features contained within the layer?

For example, I have a query created that returns the expected results, but it only returns 1000 records...the default value set when publishing the service. Is it possible to loop through the next 1000 records, and the next 1000 records after that until I have all the features within that layer?

Thanks in advance!

Re: ArcGIS Server REST API Query

PaulCrickard — Wed, 14 Jan 2015 15:42:00 GMT

If you have access to the server you can change the number returned, but I am guessing you do not.

The features only return a set # but the IDsOnly query returns everything. I ran a test and using where:1=1 I got 2,500 responses. When I did the same query using IDsOnly, I got 250,418.

Query for IDsOnly, get them all. Now you know how many features exist and the range of OBEJCTIDs. Run multiple ajax queries using the objectids parameter and comma seperatinig them using array index 0-998 then 999 to 1998.....etc.

http://server/...../0/query?f=json,objectIds=1,2,3,4,5

http://server/...../0/query?f=json,objectIds=6,7,8,9

Re: ArcGIS Server REST API Query

Jay_Gregory — Wed, 14 Jan 2015 16:05:58 GMT

I also have been waiting for a paging feature for REST queries to get around this issue, but until then there is only workarounds. Pauls solution is probably the most thorough, If you know the data and general approximate size (most later versions of Server support a getCount query which just returns the number of records) you could specify different where queries. I think an elegant solution would be some recursive function that uses a finer and finer geographic mesh to make a series of geographic intersection calls until each region returns less results than the max number allowed.

Re: ArcGIS Server REST API Query

mpboyle — Wed, 14 Jan 2015 17:00:33 GMT

Paul Crickard‌ -- this is helpful. Not the way I was hoping to do it, but it does work as a way of retrieving thousands of records.

Thanks for the tip.

Re: ArcGIS Server REST API Query

JasonTipton — Sat, 11 Dec 2021 07:16:45 GMT

Wow, thanks. I didn't know about the IDsOnly. I have done this in the past by ordering by objectid.

Here's some pseudocode

while True:
    q = getEverythingGreaterThanID(maxid)     # ordered by objectid of course
    doSomethingWithIt(q)
    if q.has_key('exceededTransferLimit') == False:
          break          # exit the loop. We're done here. No more to see here. Go Home
    maxID = getMaxID(q)
    #Repeat

I like your way better.

Re: ArcGIS Server REST API Query

XanderBakker — Wed, 14 Jan 2015 18:11:50 GMT

I have posted a simple script in this thread: https://community.esri.com/thread/118781#445572

Re: ArcGIS Server REST API Query

TanuHoque — Thu, 15 Jan 2015 17:29:09 GMT

At 10.3, two new parameters (resultOffset and resultRecordCount) are added to query operation to support pagination.

http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Query_Map_Service_Layer/02r3000000p1000000/

pls note: before you execute query operation to page through, pls check the layer's resource to find out whether it supports pagination (e.g. at 10.3, layers/tables off a FileGDB or shape files etc. don't support pagination; should work for enterprise databases). Look for 'supportsPagination' flag inside 'advancedQueryCapabilities'.

Re: ArcGIS Server REST API Query

Anonymous User — Thu, 15 Jan 2015 17:41:37 GMT

Hi Matt,

I wrote a module that can do this, I sent you a message on LinkedIn asking for your email so I can send it to you.

Caleb

Re: ArcGIS Server REST API Query

Anonymous User — Sat, 11 Dec 2021 07:16:47 GMT

Just wanted to add here I have some working code to do this. I have posted a python package on GitHub. It can export web map layers to feature classes, and keep querying features until it gets them all if the features exceed the query limit. Some sample code (package requires the requests module):

import restapi

url = 'http://some_server.com/arcgis/rest/services/folder/some_map_service/MapService'

# only use username and password if service is secured
user= 'username'
pw = 'password'
service = restapi.MapService(url, user, pw)

# export parcels layer
shp = r'C:\TEMP\output_parcels.shp'
service.layer_to_fc('parcels', shp, get_all=True)  # setting the get_all parameter to true will allow you to exceed the transfer limit

The help documentation is available here, or you can get at it from the module:

restapi.getHelp()  # launch web help

Re: ArcGIS Server REST API Query

nicogis — Tue, 02 Oct 2018 10:31:46 GMT

Tanu, ok that 'Return IDs Only' hasn't limit (maxRecordCount) but if I use resultOffset and resultRecordCount for example to do paging of oids (where 1=1 and Return IDs Only = true) seems that resultOffset and resultRecordCount are ignored. Is it for design ?

Re: ArcGIS Server REST API Query

TanuHoque — Tue, 02 Oct 2018 17:27:49 GMT

yes, that is by design that resultOffset and resultRecordCount are ignored when returnIDsOnly=True.

not sure why you'd want to page thru objectids, but if you really need to do that you can submit a query where outFields=objectid & returnIDsOnly=False along with pagination parameters (i.e. resultOffset and/or resultRecordCount).

hope this helps.

Re: ArcGIS Server REST API Query

nicogis — Tue, 02 Oct 2018 18:47:57 GMT

I have written a code that use list of oids and I have added the condition of paging but I haven't modified my function return oids (get_object_ids). But if returnIdsOnly is ignored with offset no problem: I have removed the condition in code:

Fixed services that support paging · nicogis/DownloadService@ab17f3f · GitHub

Tanu, but you could receive this error with a simple request rest query with returnIdsOnly ( there is no limit on the number of object IDs returned in the ID array response ) so can be useful do paging of oids

I have generated about 8000000 oids

In some server with high number of services I had 'Unable to process request. Error handling service request :Cannot obtain a free instance.; nested exception is: java.lang.OutOfMemoryError: GC overhead limit exceeded'. ( GC overhead limit exceeded – Plumbr ) so I had also the need change 'app server maximum heap size' for allocate more memory.

Re: ArcGIS Server REST API Query

KadeSmith — Thu, 19 Dec 2019 19:20:57 GMT

In case anyone else comes across this post, the get_all=True has been updated to exceed_limit=True on line 12 of the code Caleb posted.