How can I query more than 10,000 users at a time from my Portal?

EricaPfister

I need to pull a list of all our Portal users. There are a couple of Python tools we have built that use this full list, and then iterate through each user to check for various characteristics.

However, now that our Portal has over 11k users we have noticed that the previous code (snippet below) is no longer pulling the complete list. It pulls only 10,000 records, even though the max_users parameter is set to 99,999.

user_list = gis.users.search(query='!admin', max_users=99999)

https://developers.arcgis.com/python/api-reference/arcgis.gis.toc.html#arcgis.gis.UserManager.search does not mention any sort of start parameter which I could potentially use to paginate, just the max_users option. Is there something that I can do which will let me query that full list of more than 10,000 users? I can switch over to using the REST API if I absolutely must, but it's surprising/disappointing to me that I can do almost anything else through the ArcGIS API for Python.

LongDinh

Hi @EricaPfister,

It seems that arcgis module has not implemented pagination yet. According to the User Search API, you can search from an index using the start parameter which is used to paginate the search result.

The API reference notes are a bit vague, but to search for 100,000 users or until the max users is reached, it would go something like this:

 

 

 

 

import requests

# Get your token
token = getToken() # A function to retrieve an Admin Portal Token to access the API

# Payload init values
query = ""
start = 1
num = 50
sortField = 'username'
returned_total = 1000

total_users_to_return = 100000

user_search_api = "https://.../arcgis/sharing/rest/communitiy/user"

# Search through entire portal for users until the total number returend
next_start = True
user_count = 0

results = []
_next_start = None
while next_start and user_count <=total_users_to_return:
    if _next_start is not None:
        start = _next_start
    payload = {
        'q':query,
        'start':start,
        'sortField': sortField,
        'num':num,
        'f':'json',
        'token':token
    }
    try:
        response = requests.post(
            user_search_api,
            data=payload
        )

        resp_json = response.json()
        # Add the result to the results
        if resp_json.get('results'):
            results += resp_json.get('results')

        # Check while conditions
        user_count += resp_json.get('total')

        if resp_json.get('nextStart'):
            _next_start = resp_json.get('nextStart')
        else:
            # Stop the loop
            next_start = False
        
    except Exception as e:
        print (e)
        print (f"Failed to get start index: {start}")
        continue
 

print (f"Retrieved {len(results)} users.")