topic Re: Trying to iterate through all my feature services - issue with advanced_search? in ArcGIS API for Python Questions

Trying to iterate through all my feature services - issue with advanced_search?

SimonGIS — Mon, 31 May 2021 14:20:49 GMT

As an admin for ArcGIS Online, I want to iterate through all of my hosted feature services so that I can check a few things.

Organisation has over 10,000 items
Having issues between the count of records returned from what appears to be the same query with search and advanced_search

from arcgis.gis import GIS # connect to the portal gis = GIS(profile='my_prof', expiration=9999) #PROD # Create two lists to store results in adv_S_list = [] search_list = [] # Count with standard search, maxes out at 10000 search_items = gis.content.search(query="*", item_type="Feature Service", max_items = 10000) for i in search_items: search_list.append((i.title, i.type)) # Count from regular search print("#FS Items found using search: " + str(len(search_items))) #returns 3993, this seems correct from looking at items in search_list # get a count using advanced_search count_items = gis.content.advanced_search( query = "orgid:vHnIGBHHqDR6y0CR and type:'Feature Service'", max_items = 70000, return_count = True ) # Count from advanced_search print("#FS Items found using advanced_search: " + str(count_items)) # returns 481 # paginate through using advanced_search and get a count start = 0 len_sr = 100 while len_sr == 100: all_items = gis.content.advanced_search( query = "orgid:vHnIGBHHqDR6y0CR and type:'Feature Service'", start = start, max_items = 100 ) list_items = all_items.get('results') len_sr = len(list_items) start = start + 100 for i in list_items: adv_S_list.append((i.title, i.type)) # Count from advanced_search pagination approach: print("#FS Items found using advanced_search + pagination: " + str(len(adv_S_list))) # returns 482

What is the right approach to correctly loop through all the reocds from a feature service?

Re: Trying to iterate through all my feature services - issue with advanced_search?

Kevin_McIntyre — Mon, 31 May 2021 13:48:41 GMT

Are you querying Feature Layers or Feature Services or both? Your code originally searches for Feature Layers but then searches for Feature Services. You're going to get different counts because you're searching for different items essentially.

Re: Trying to iterate through all my feature services - issue with advanced_search?

SimonGIS — Mon, 31 May 2021 14:24:12 GMT

Hey @Kevin_McIntyre - good spot.

End goal: Return all hosted feature services in ArcGIS Online.

With advanced_search I believe I am limited to the REST Search reference which lists feature service as a type.

I have edited and re-run my code above to match the standard search type to the advanced_search type:

search_items = gis.content.search(query="*", item_type="Feature Service", max_items = 10000)

The counts are similar (suspect some people have deleted items since last running code):

#FS Items found using search: 3993
#FS Items found using advanced_search: 480
#FS Items found using advanced_search + pagination: 481

Are you able to run this code against an organisation at your end with a large number of items?

Re: Trying to iterate through all my feature services - issue with advanced_search?

Kevin_McIntyre — Mon, 31 May 2021 15:25:38 GMT

Just tried it on my organization, and while we don't have 10,000 items, I was getting a similar discrepancy between the standard search and advanced_search methods.

However, I altered the syntax of the advanced_search query parameter from:

query = "orgid:<OrgID> and type:'Feature Service'"

query = "orgid:<OrgID> AND type:'Feature Service'"

Changing the logical operator to uppercase in the query worked on my end and caused the two search methods to agree with each other.

Re: Trying to iterate through all my feature services - issue with advanced_search?

SimonGIS — Mon, 31 May 2021 23:44:19 GMT

Yes! Thanks for spotting and testing that @Kevin_McIntyre - I had a feeling it would be something fairly trivial.

I still have some discrepancy, but I can work with that. Case closed. Thank you

#FS Items found using search: 3993
#FS Items found using advanced_search: 480
#FS Items found using advanced_search + pagination: 3999