Limit for number of items returned in content.advanced_search?

Question

Hello,

Is there a hard limit on the number of items that can be returned by ContentManager's advanced search function? Our organization has more than 10,000 items. I've been querying the API for items in groups of 100 and paging through them. Each query result shows the total number of items as 10,000, but at last check via a generated report on our items, we have 11,577 items. When the paged query reaches start=10,001, the results start returning empty items and the query shows as "None". This seems to continue on forever. Is this expected behavior? Is there a hard limit of 10,000 items for advanced_search?

mikeytes · Answer

Did you ever find a conclusive answer @MikeBijou2? What we've discovered is if the query results in more than 10,000 items, then it will never return more than 10,000 items...even if you paginate! However, if you alter the query to limit the amount of items returned, then you have a shot at returning all your organization's items. For example you can do something like this...

import datetime as dt

all_items = []
for year in range(2026, 2008, -1):
    start_ob = dt.datetime(month=1, day=1, year=year)
    start = int(start_ob.timestamp() * 1000)

end_ob = dt.datetime(month=1, day=1, year=year+1)
    end = int(end_ob.timestamp() * 1000)
    all_items.extend(gis.content.advanced_search(query=f"accountid:{gis.properties.id} AND modified:[{start} TO {end}]", max_items=10000)["results"])

Also, you don't have to use the `modified` property. You can use username, item type, etc.

Still... I would love an answer on why the max is 10000.

MikeBijou2 · Answer

Hello Johannes,

I actually do use the max_items parameter; but this doesn't affect the total number of items that can be returned by the query. I had to set max_items to 100, because using a number higher than that, the API will still break the query into groups of 100 but send multiple requests simultaneously. Previously, I used max_items = 10000 and my account got blocked for sending too many calls to the API. No matter what I set max_items to however, the query result will still only show a total amount of 10,000 items.

This is the code in question:

while True:
        logger.info(f"items search {ctr} {root_url}")
        item_result = gis.content.advanced_search(
            f"orgid:{org_id}", start=ctr, max_items=100
        )
        time.sleep(3)
        if item_result["results"]:
            ctr += 100
            items.extend(item_result["results"])
        else:
            break

JohannesLindner · Answer

The advanced_search method of ContentManager has a max_items argument, that sadly isn't documented in the parameters table. Setting that to a high value might help.

from arcgis.gis import GIS

gis = GIS("home")
users = gis.users.search(max_users=1000)
query = " OR ".join([f"owner:{u.username}" for u in users])

results1 = gis.content.advanced_search(query)["results"]
results2 = gis.content.advanced_search(query, max_items = 200000)["results"]

len(results1)  # 100
len(results2)  # 190189