This is a really pesky issue. I have confirmed that my spatial query should return 1931 records (confirmed with local data via ArcGIS Pro as well as REST query with returnCountOnly set to true).
However, using a utility function that paginates through results using the exceededTransferLimit flag, I get back 2370 results.
My function submits an initial query which returns 1931 results, but the exceededTransferLimit flag is set to true, so it submits a second time with the resultOffset set to 1931 and gets back 439 features with no exceededTransferLimit key (not even set to false - it's just not included in the response) so my final result set is 2370 records instead of 1931.
The python API only returns 1931, so that seems works, but I have users just using Python requests. Is there something I'm implementing incorrectly?
def paginate_queries(query_url:str, params:object, features:list=[])->pd.DataFrame:
if len(features)>0:
params['resultOffset']=len(features)
result = requests.post(query_url, params=params).json()
print(f"{len(result['features'])} features returned")
newfeatures = features + result['features']
if ('exceededTransferLimit' in result.keys()) and (result['exceededTransferLimit']):
return paginate_queries(query_url, params, features=newfeatures)
return pd.json_normalize(newfeatures)