nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

pagination issue -- resultRecordCount and resultOffset

simoxu

I applied a geometry filter in the query to a ArcGIS Server Map Service (query enabled), when the ResultOffset is set to 0 and resultRecordCount is set to 500, it only returns 36 records

If I remove these parameters in the query, it returns all the features intersects with the geometry:

This only happens to the ArcGIS Server Service, if the service is hosted in the ArcGIS Enterprise Portal, it works OK.

Any idea what's happening?

Thanks you in advance.

please note that I posted the same question in the ArcGIS REST API Community a couple of days ago, but I am thinking it might also be a ArcGIS Server topic, as this happens on a ArcGIS Server Map/ Feature Service, but it works fine for a hosted service (I know the service is hosted on a federated server...), I am wondering it might have something to do with my ArcGIS Server settings. if it is true.

Encuentra más publicaciones etiquetadas con

Accepted answers

ahoque

Hi Simo,

That is expected behavior.

In short, it is due to how spatial filters are handled in ArcGIS Enterprise Map and Feature Services. The long story is complicated, and I don't think I'd be able to explain this very clearly here without making you more confused.

You should check for 'exceedTransferLimit' flag. This is what we have in REST API help.

The exceededTransferLimit property is now included in the JSON response when paging through a query result with the resultOffset and resultRecordCount parameters. When exceededTransferLimit is true, it indicates there are more query results and you can continue to page through the results. When exceededTransferLimit is false, it indicates that you have reached the end of the query results.

Hope this helps.
Tanu

All comments

SteveWealands

Seems that this issue of 'less than the expected number of records' being returned is a known bug: https://support.esri.com/en-us/bug/in-arcgis-server-if-the-values-for-result-offset-and-re-bug-000120308

I have been paginating data and relying on the 'exceededTransferLimit' flag to know when to request for more. Have found that I need to increment the resultOffset by the resultRecordCount each time to get close to the right result.

The only thing I am still seeing is that when I request a page of data that isn't full (e.g. request 1000 results but due to the known bug only 995 are returned with the flag raised), the subsequent page I request has the last feature from the previous page on it again. Crazy. See the issue here for more info:

https://gis.stackexchange.com/questions/499944/esri-rest-query-with-spatial-intersect-inconsistent-results

Can anyone else reproduce that issue on their feature server?

Jay_Gregory

This is a really pesky issue. I have confirmed that my spatial query should return 1931 records (confirmed with local data via ArcGIS Pro as well as REST query with returnCountOnly set to true).

However, using a utility function that paginates through results using the exceededTransferLimit flag, I get back 2370 results.

My function submits an initial query which returns 1931 results, but the exceededTransferLimit flag is set to true, so it submits a second time with the resultOffset set to 1931 and gets back 439 features with no exceededTransferLimit key (not even set to false - it's just not included in the response) so my final result set is 2370 records instead of 1931.

The python API only returns 1931, so that seems works, but I have users just using Python requests. Is there something I'm implementing incorrectly?

def paginate_queries(query_url:str, params:object, features:list=[])->pd.DataFrame:
    if len(features)>0:
        params['resultOffset']=len(features)
    result = requests.post(query_url, params=params).json()
    print(f"{len(result['features'])} features returned")
    newfeatures = features + result['features']
    if ('exceededTransferLimit' in result.keys()) and (result['exceededTransferLimit']):
        return paginate_queries(query_url, params, features=newfeatures)
    return pd.json_normalize(newfeatures)

simoxu

You might find this post helpful.

https://github.com/Esri/esri-leaflet/issues/786

AndyWhitaker1

@simoxu, was this problem fixed? I seem to be experiencing a similar issue in the ArcGIS API for JavaScript.

simoxu

Thanks.

I've contacted ESRI local support for this issue to be escalated.

ahoque

If this is an expected behavior, I am just curious why the hosted service in AGOL and ArcGIS portal do not have this issue, as we all know behind the scenes, the same ArcGIS server is doing the heavy lifting.

That is because technically 'hosted services' and 'enterprise map and feature services' are two different implementation. There are some age old specifications that need to be honored in enterprise implementation of map and feature service that leads us to this known issue.

More importantly, this expected ArcGIS server behaviour is causing issue in the APIs (at least I have personally experienced the issue in ArcGIS Python API). I dived into the API source code and found, the code checks for the number of records returned, if it is less than the resultRecordCount or the MaxRecordCount (whichever is less) it will assume all the records has been retrieved and stop querying the service.

Thanks for noticing this. That seems something that might need to fix in ArcGIS Python API. Would you mind contact Esri Support and have an issue logged in.

Thanks.

Tanu

simoxu

Hi Tanu,

Thanks for your answer to the question.

If this is an expected behavior, I am just curious why the hosted service in AGOL and ArcGIS portal do not have this issue, as we all know behind the scenes, the same ArcGIS server is doing the heavy lifting.

More importantly, this expected ArcGIS server behaviour is causing issue in the APIs (at least I have personally experienced the issue in ArcGIS Python API). I dived into the API source code and found, the code checks for the number of records returned, if it is less than the resultRecordCount or the MaxRecordCount (whichever is less) it will assume all the records has been retrieved and stop querying the service.

Many thanks

Simo

ahoque

Hi Simo,

That is expected behavior.

You should check for 'exceedTransferLimit' flag. This is what we have in REST API help.

The exceededTransferLimit property is now included in the JSON response when paging through a query result with the resultOffset and resultRecordCount parameters. When exceededTransferLimit is true, it indicates there are more query results and you can continue to page through the results. When exceededTransferLimit is false, it indicates that you have reached the end of the query results.

Hope this helps.
Tanu

Únete a la Conversación

Los miembros registrados pueden publicar, seguir actualizaciones y más. ¿Nuevo aquí? Regístrate gratis.

Enlaces Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.