Limit for number of items returned in content.advanced_search?

662
2
01-27-2023 12:57 PM
MikeBijou2
New Contributor II

Hello,

Is there a hard limit on the number of items that can be returned by ContentManager's advanced search function? Our organization has more than 10,000 items. I've been querying the API for items in groups of 100 and paging through them. Each query result shows the total number of items as 10,000, but at last check via a generated report on our items, we have 11,577 items. When the paged query reaches start=10,001, the results start returning empty items and the query shows as "None". This seems to continue on forever. Is this expected behavior? Is there a hard limit of 10,000 items for advanced_search?

0 Kudos
2 Replies
JohannesLindner
MVP Frequent Contributor

The advanced_search method of ContentManager has a max_items argument, that sadly isn't documented in the parameters table. Setting that to a high value might help.

 

from arcgis.gis import GIS

gis = GIS("home")
users = gis.users.search(max_users=1000)
query = " OR ".join([f"owner:{u.username}" for u in users])

results1 = gis.content.advanced_search(query)["results"]
results2 = gis.content.advanced_search(query, max_items = 200000)["results"]

len(results1)  # 100
len(results2)  # 190189

 


Have a great day!
Johannes
0 Kudos
MikeBijou2
New Contributor II

Hello Johannes,

I actually do use the max_items parameter; but this doesn't affect the total number of items that can be returned by the query. I had to set max_items to 100, because using a number higher than that, the API will still break the query into groups of 100 but send multiple requests simultaneously. Previously, I used max_items = 10000 and my account got blocked for sending too many calls to the API. No matter what I set max_items to however, the query result will still only show a total amount of 10,000 items.

This is the code in question: 

while True:
        logger.info(f"items search {ctr} {root_url}")
        item_result = gis.content.advanced_search(
            f"orgid:{org_id}", start=ctr, max_items=100
        )
        time.sleep(3)
        if item_result["results"]:
            ctr += 100
            items.extend(item_result["results"])
        else:
            break

  

0 Kudos