Feature Layer query can't return all the matching records

[simoxu]

I am using 1.7.0 version of the Python API.

I have a fairly large feature layer published in our ArcGIS Portal 10.7, roughly 1 million records in it.

When I do a query on this layer, and only return the count only, I'll get the count 1329 which is correct. 

But when I try to return the actual records, it only returns 99 records despite that I've set return_all_records to True.

Anyone know why?  and how can I fix it?

Many thanks.

[JoshuaBixby]

At this point, it is unclear whether this is a REST API or ArcGIS API for Python issue.  Have you tried directly querying the REST API to see what results you get?

[simoxu]

Thanks for the response.

I tested the same spatial filter in REST API, it seems returning all the records.

So it is clear to me the issue is in the python API. 

[JoshuaBixby]

What if you check the records in the feature set before converting to data frame?  That might tell you whether it is a feature set issue or a data frame issue.

[simoxu]

Yes, I tried that. the query does not return all the features intersecting with the input geometry.

This only happens when the geometry filter is being used. there's no problem with the attribute filter using the where parameter.

[JoshuaBixby]

I wonder if the results are correct in the ArcGIS API for Python but they are based on something happening in translation of your geometry filter from Python to REST.  What is your geometry filter? 

[simoxu]

Hi Joshua

The geometry filter is a ploygon that returns all the land parcels that intersect with it.

Here is an update on what I've done:

I exported JSON of this geometry and tried it in the REST API, it works perfectly.

In Python API, the geometry works fine if only IDs, or counts are returned. so my current workaround is to return all the IDs then query the same layer again using the where parameter to return all the features, which works fine, but apparently this is not the ideal way to do it.

I also tried this in a standalone script, same issue. So it's not a Jupyter notebook bug either. so far I can pretty much isolate it to ArcGIS Python API. I just need ESRI to confirm my guessing. 

I've send ESRI a support request,  and I'll keep this post updated when I hear from them.

Cheers

Simo

[JoshuaBixby]

Nice work, and sometimes a workaround is the best, most practical path forward. Can you reproduce the behavior using a public-facing service and some arbitrary polygon?  If so, please share.  I am guessing it has to do with the geometry information is getting translated from Python to REST.

I will be interested what you hear back.

[simoxu]

No worries. 

I also tried the same query on a large point layer (the centroids of the features in the original polygon layer), and it works fine. so it happens on polygon layers, I haven't tested polygon lines which I don't use very often. 

Hope we can get the attention from a member in the Dev Team? John Yaist‌

[JohnYaist1]

I'll look through this today, simo xu.  Do you see the same results with 1.7.1?