Trying to iterate through all my feature services - issue with advanced_search?

[SimonGIS]

As an admin for ArcGIS Online, I want to iterate through all of my hosted feature services so that I can check a few things.

• Organisation has over 10,000 items
• Having issues between the count of records returned from what appears to be the same query with search and advanced_search

from arcgis.gis import GIS

# connect to the portal
gis = GIS(profile='my_prof', expiration=9999)  #PROD

# Create two lists to store results in
adv_S_list = []
search_list = []

# Count with standard search, maxes out at 10000 
search_items = gis.content.search(query="*", item_type="Feature Service", max_items = 10000)  

for i in search_items:
    search_list.append((i.title, i.type))

# Count from regular search
print("#FS Items found using search: " + str(len(search_items)))  #returns 3993, this seems correct from looking at items in search_list

# get a count using advanced_search
count_items = gis.content.advanced_search(
    query = "orgid:vHnIGBHHqDR6y0CR and type:'Feature Service'",
    max_items = 70000,
    return_count = True    
    )

# Count from advanced_search
print("#FS Items found using advanced_search: " + str(count_items)) # returns 481
      

# paginate through using advanced_search and get a count
start = 0
len_sr = 100



while len_sr == 100:
    all_items = gis.content.advanced_search(
        query = "orgid:vHnIGBHHqDR6y0CR and type:'Feature Service'",
        start = start,
        max_items = 100
        )
    
    list_items = all_items.get('results')
    len_sr = len(list_items)
    start = start + 100
    for i in list_items:
        adv_S_list.append((i.title, i.type))

# Count from advanced_search pagination approach:    
print("#FS Items found using advanced_search + pagination: " + str(len(adv_S_list)))  # returns 482

 What is the right approach to correctly loop through all the reocds from a feature service?

[Kevin_McIntyre]

Are you querying Feature Layers or Feature Services or both? Your code originally searches for Feature Layers but then searches for Feature Services. You're going to get different counts because you're searching for different items essentially. 

[SimonGIS]

Hey @Kevin_McIntyre - good spot. 

End goal: Return all hosted feature services in ArcGIS Online.

With advanced_search I believe I am limited to the REST Search reference which lists feature service as a type.

I have edited and re-run my code above  to match the standard search type to the advanced_search type:

search_items = gis.content.search(query="*", item_type="Feature Service", max_items = 10000)

The counts are similar (suspect some people have deleted items since last running code):

#FS Items found using search: 3993
#FS Items found using advanced_search: 480
#FS Items found using advanced_search + pagination: 481

Are you able to run this code against an organisation at your end with a large number of items?

[Kevin_McIntyre · Accepted solution]

Just tried it on my organization, and while we don't have 10,000 items, I was getting a similar discrepancy between the standard search and advanced_search methods. 

However, I altered the syntax of the advanced_search query parameter from:

query = "orgid:<OrgID> and type:'Feature Service'"

to 

query = "orgid:<OrgID> AND type:'Feature Service'"

Changing the logical operator to uppercase in the query worked on my end and caused the two search methods to agree with each other.

[SimonGIS]

Yes!  Thanks for spotting and testing that @Kevin_McIntyre - I had a feeling it would be something fairly trivial. 

I still have some discrepancy, but I can work with that.  Case closed. Thank you

#FS Items found using search: 3993
#FS Items found using advanced_search: 480
#FS Items found using advanced_search + pagination: 3999