pagination issue -- resultRecordCount and resultOffset

[simoxu]

I applied a geometry filter in the query to a ArcGIS Server Map Service (query enabled), when the ResultOffset is set to 0 and resultRecordCount is set to 500, it only returns 36 records

If I remove these parameters in the query, it returns all the features intersects with the geometry:

This only happens to the ArcGIS Server Service, if the service is hosted in the ArcGIS Enterprise Portal, it works OK.

Any idea what's happening?

Thanks you in advance.

please note that I posted the same question in the ArcGIS REST API Community a couple of days ago, but I am thinking it might also be a ArcGIS Server topic,  as this happens on a ArcGIS Server Map/ Feature Service, but it works fine for a hosted service (I know the service is hosted on a federated server...), I am wondering it might have something to do with my ArcGIS Server settings. if it is true.

[TanuHoque · Accepted solution]

Hi Simo,

That is expected behavior.

In short, it is due to how spatial filters are handled in ArcGIS Enterprise Map and Feature Services. The long story is complicated, and I don't think I'd be able to explain this very clearly here without making you more confused.

You should check for 'exceedTransferLimit' flag. This is what we have in REST API help.

The exceededTransferLimit property is now included in the JSON response when paging through a query result with the resultOffset and resultRecordCount parameters. When exceededTransferLimit is true, it indicates there are more query results and you can continue to page through the results. When exceededTransferLimit is false, it indicates that you have reached the end of the query results.

Hope this helps.
Tanu

[simoxu]

Hi Tanu,

Thanks for your answer to the question.

If this is an expected behavior, I am just curious why the hosted service in AGOL and ArcGIS portal do not have this issue, as we all know behind the scenes, the same ArcGIS server is doing the heavy lifting.

More importantly, this expected ArcGIS server behaviour is causing issue in the APIs (at least I have personally experienced the issue in ArcGIS Python API). I dived into the API source code and found, the code checks for the number of records returned, if it is less than the resultRecordCount or the MaxRecordCount (whichever is less) it will assume all the records has been retrieved and stop querying the service.

Many thanks

Simo

[TanuHoque]

If this is an expected behavior, I am just curious why the hosted service in AGOL and ArcGIS portal do not have this issue, as we all know behind the scenes, the same ArcGIS server is doing the heavy lifting.

That is because technically 'hosted services' and 'enterprise map and feature services' are two different implementation. There are some age old specifications that need to be honored in enterprise implementation of map and feature service that leads us to this known issue.

More importantly, this expected ArcGIS server behaviour is causing issue in the APIs (at least I have personally experienced the issue in ArcGIS Python API). I dived into the API source code and found, the code checks for the number of records returned, if it is less than the resultRecordCount or the MaxRecordCount (whichever is less) it will assume all the records has been retrieved and stop querying the service.

Thanks for noticing this. That seems something that might need to fix in ArcGIS Python API. Would you mind contact Esri Support and have an issue logged in.

Thanks.

Tanu

[simoxu]

Thanks.

I've contacted ESRI local support for this issue to be escalated.

[AndyWhitaker1]

@simoxu, was this problem fixed?  I seem to be experiencing a similar issue in the ArcGIS API for JavaScript.

[simoxu]

You might find this post helpful.

https://github.com/Esri/esri-leaflet/issues/786

[Jay_Gregory]

This is a really pesky issue.  I have confirmed that my spatial query should return 1931 records (confirmed with local data via ArcGIS Pro as well as REST query with returnCountOnly set to true). 

However, using a utility function that paginates through results using the exceededTransferLimit flag, I get back 2370 results. 

My function submits an initial query which returns 1931 results, but the exceededTransferLimit flag is set to true, so it submits a second time with the resultOffset set to 1931 and gets back 439 features with no exceededTransferLimit key (not even set to false - it's just not included in the response) so my final result set is 2370 records instead of 1931. 

The python API only returns 1931, so that seems works, but I have users just using Python requests.  Is there something I'm implementing incorrectly?

def paginate_queries(query_url:str, params:object, features:list=[])->pd.DataFrame:
    if len(features)>0:
        params['resultOffset']=len(features)
    result = requests.post(query_url, params=params).json()
    print(f"{len(result['features'])} features returned")
    newfeatures = features + result['features']
    if ('exceededTransferLimit' in result.keys()) and (result['exceededTransferLimit']):
        return paginate_queries(query_url, params, features=newfeatures)
    return pd.json_normalize(newfeatures)