Feature Layer query can't return all the matching records

[simoxu]

I am using 1.7.0 version of the Python API.

I have a fairly large feature layer published in our ArcGIS Portal 10.7, roughly 1 million records in it.

When I do a query on this layer, and only return the count only, I'll get the count 1329 which is correct. 

But when I try to return the actual records, it only returns 99 records despite that I've set return_all_records to True.

Anyone know why?  and how can I fix it?

Many thanks.

[JohnYaist1]

Hi simo xu 

I'm not seeing what you're describing using that last couple releases of the Python API. I tried this on a couple different layers, one small one, one larger one (not a huge feature layer, but the features contain a large number of vertices).

Do you see this on any other polygon layer with an arbitrary geometry filter? Do you have a public-facing layer as Joshua Bixby suggests? Can you try explicitly adding the where clause to see if that works?

"where='1=1'"‍

[simoxu]

Hi John Yaist‌

Thank you for looking into this.

I tried the same code in a 1.7.1 version, it gives me same result.

Here is the environment:

The geometry of the filter is quite complicated as shown below:

val_flayer is the cadastral feature layer.

The feature service I am using is for internal use and behind our organisational firewall, which is unfortunate, otherwise,  it would be easy to reproduce the issue. 

Thanks

[simoxu]

John Yaist‌

Update:

This strange behavior only happens when Feature Service being queried is published to the Federated ArcGIS server directly --  this is my case.

I tried to upload the same FGDB zip file to our ArcGIS Portal and publish it there as a hosted service, it works fine. 

[simoxu]

It has something to do with query pagination.

when I monitor the the REST requests it sent to the server, it sends two requests:

1. get the the count of the records

2. get the actual features with  resultRecordCount paramter sets to the result of the first request.

This is very logical and should work, but it doesn't for the feature service directly published to the federated server.

if I remove the resultRecordCount parameter in the REST request in the second step. it returns all the features... 

[JohnYaist1]

Hi simo xu‌

What version of Server are you publishing this service to? 

Can you outline exactly how the service is "published to the Federated ArcGIS server directly?" I want to follow the exact steps to test on a service published in the exact same manner.

[simoxu]

It's proven that the actual differences are in the hosted service and non-hosted service. hosted service works regardless point layer or polygon layer. and I can reproduce the error using REST API. so this may not be a ArcGIS Python API bug. The issue is at a lower level I am afraid.

[simoxu]

Hi John Yaist‌

This is very likely an ArcGIS Python API bug.

In the API source code for the query function (layer.py, Line 650), it does not test the exceededTransferLimit property in the response, the code assumes the query always return the specified number of records when possible. This is not the case for spatial filter....

exceededTransferLimit  was introduced in ArcGIS Server 10.3.1, please see the details in the following link

In some cases when using the resultOffset and resultRecordCount parameters, the exceededTransferLimit property may be included in the query results even though the value specified in the resultRecordCount has not been exceeded. This is due to internal spatial index filtering of the query results. For this reason, you should always rely on the exceededTransferLimit property to determine if you should page through results rather than relying on the number of results returned from each page. In some extreme cases, zero results can be returned but the exceededTransferLimit property will be returned. In these cases, you should continue paging through your results until exceededTransferLimit is no longer returned.

Query (Feature Service/Layer)—ArcGIS REST API: Services Directory | ArcGIS for Developers 

Hope this makes sense.

Regards

Simo

[JohnYaist1]

Thanks for all this information, simo xu‌. To get an official bug report logged I suggest contacting support who can document the specific workflow.  I'll look into investigating on a service published directly to a federated server rather than a hosted one and see what I can reproduce. 

[simoxu]

Thanks John Yaist‌, I've been talking with ESRI local support and just urged them to escalate it to ESRI US.

[simoxu · Accepted solution]

BUG-000129661: Querying a feature layer from a referenced feature service with a geometry filter returns an incomplete results using the ArcGIS API for Python