ArcGIS Server REST API Query

[mpboyle]

I'm wondering if it's possible to create a script that loops through an ArcGIS Server REST API query in order to retrieve all features contained within the layer?

For example, I have a query created that returns the expected results, but it only returns 1000 records...the default value set when publishing the service.  Is it possible to loop through the next 1000 records, and the next 1000 records after that until I have all the features within that layer?

Thanks in advance!

[PaulCrickard]

If you have access to the server you can change the number returned, but I am guessing you do not.

The features only return a set # but the IDsOnly query returns everything. I ran a test and using where:1=1 I got 2,500 responses. When I did the same query using IDsOnly, I got 250,418.

Query for IDsOnly, get them all. Now you know how many features exist and the range of OBEJCTIDs. Run multiple ajax queries using the objectids parameter and comma seperatinig them using array index 0-998 then 999 to 1998.....etc.

http://server/...../0/query?f=json,objectIds=1,2,3,4,5

http://server/...../0/query?f=json,objectIds=6,7,8,9

[mpboyle]

Paul Crickard‌ -- this is helpful.  Not the way I was hoping to do it, but it does work as a way of retrieving thousands of records.

Thanks for the tip.

[JasonTipton]

Wow, thanks. I didn't know about the IDsOnly. I have done this in the past by ordering by objectid.

Here's some pseudocode

while True:
    q = getEverythingGreaterThanID(maxid)     # ordered by objectid of course
    doSomethingWithIt(q)
    if q.has_key('exceededTransferLimit') == False:
          break          # exit the loop. We're done here. No more to see here. Go Home
    maxID = getMaxID(q)
    #Repeat

I like your way better.

[Jay_Gregory]

I also have been waiting for a paging feature for REST queries to get around this issue, but until then there is only workarounds.  Pauls solution is probably the most thorough, If you know the data and general approximate size (most later versions of Server support a getCount query which just returns the number of records) you could specify different where queries.  I think an elegant solution would be some recursive function that uses a finer and finer geographic mesh to make a series of geographic intersection calls until each region returns less results than the max number allowed. 

[TanuHoque]

At 10.3, two new parameters (resultOffset and resultRecordCount) are added to query operation to support pagination.

http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Query_Map_Service_Layer/02r3000000p1...

pls note: before you execute query operation to page through, pls check the layer's resource to find out whether it supports pagination (e.g. at 10.3, layers/tables off a FileGDB or shape files etc. don't support pagination; should work for enterprise databases). Look for 'supportsPagination' flag inside 'advancedQueryCapabilities'.

[nicogis]

Tanu, ok that 'Return IDs Only' hasn't limit (maxRecordCount) but if I use resultOffset and resultRecordCount for example to do paging of oids (where 1=1 and Return IDs Only = true) seems that resultOffset and resultRecordCount are ignored. Is it for design ?

[TanuHoque]

yes, that is by design that resultOffset and resultRecordCount are ignored when returnIDsOnly=True.

not sure why you'd want to page thru objectids, but if you really need to do that you can submit a query where  outFields=objectid & returnIDsOnly=False along with pagination parameters (i.e. resultOffset and/or resultRecordCount).

hope this helps.

[nicogis]

I have written a code that use list of oids and I have added the condition of paging but I haven't modified my function return oids (get_object_ids). But if returnIdsOnly is ignored with offset no problem: I have removed the condition in code:

Fixed services that support paging · nicogis/DownloadService@ab17f3f · GitHub   

Tanu, but you could receive this error with a simple request rest query with returnIdsOnly ( there is no limit on the number of object IDs returned in the ID array response ) so can be useful do paging of oids

I have generated about 8000000 oids

 In some server with high number of services I had 'Unable to process request. Error handling service request :Cannot obtain a free instance.; nested exception is: java.lang.OutOfMemoryError: GC overhead limit exceeded'. ( GC overhead limit exceeded – Plumbr ) so I had also the need change 'app server maximum heap size' for allocate more memory.

[XanderBakker]

I have posted a simple script in this thread: https://community.esri.com/thread/118781#445572