A quick overview of group membership in your AGOL organization

2343
10
03-21-2019 02:35 PM
Egge-Jan_Pollé
MVP Regular Contributor
4 10 2,343

Do you recognize this?

  • "Over time our ArcGIS Online organization has grown. We have quite a few users now and a considerable number of groups has been created. Can you please provide me with a quick overview of who is member of which group?"
  • "Yes, I can."
  • "OK - and please note: not only quick, but also in Excel. Is that possible?"
  • "Eh... yeah, that is also possible."

We are going to use the ArcGIS API for Python with groups.search() and users.search() to get an overview of our organization. Next we compile a matrix - users along the y-axis and groups along the x-axis - with a 1 or a 0 to indicate wether the user is a member of the group or not. This matrix is being written to a CSV file.

And then the boss should be able to import this CSV into a colorful and manageable spreadsheet. With or without a little help.

Does the script below work for you? Just 'like' or 'share' if it does. And if you want to, you can just open your Jupyter Notebook to try it line by line.

## ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
## ArcGIS Online Management Information
## Script: agol_group_membership.py
## Goal: to create an overview of group membership in your ArcGIS Online organisation
## Author: Egge-Jan Polle - Tensing GIS Consultancy
## Date: August 3, 2018
## ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#
# This script should be run within a specific ArcGIS/Python environment using the batch file below
# (This batch file comes with the installation of ArcGIS Pro)
# "C:\Program Files\ArcGIS\Pro\bin\Python\scripts\propy.bat" agol_group_membership.py
#
import csv,os, sys
from arcgis.gis import GIS
from provide_credentials import provide_credentials

print('===================')
print('The script that is running: ' + __file__)
print('First you have to log in to ArcGIS Online')

# Log in
username, password = provide_credentials()
my_agol = GIS("https://www.arcgis.com", username, password)

print ("Start: "+datetime.datetime.today().strftime('%c'))

## Get all groups
my_groups = my_agol.groups.search()
## Optionally: have a look at all groups
#my_groups
## Optionally: count the number of groups
#len(my_groups)

## Get all users
my_users = my_agol.users.search(max_users = 350) # The default of max_users = 100, so increase it if you have more
## Optionally: have a look at all users
#my_users
## Optionally: count the number of users
#len(my_users)

## Create a list with all group titles
my_group_titles = []
for my_group in my_groups:
    my_group_titles.append(my_group.title)

## Create a list with field names
fieldnames = []
fieldnames = ['USERNAME','EMAIL']
for title in my_group_titles:
    fieldnames.append(title)
## Optionally: have a look at the field names
#fieldnames

## Create a CSV file with a matrix of the groups with their members
today = datetime.datetime.today().strftime('%Y%m%d')
fname = 'AGOL_Group_Membership'+today+'.csv'
try:
    os.remove(fname)
except OSError:
    pass
outfile = open(fname, 'a')
writer = csv.DictWriter(outfile, delimiter = ';', lineterminator='\n', fieldnames=fieldnames)
writer.writeheader()
## Add for each user the full name and email and for each group a 1 or 0, depending on group membership
for user in my_users:
    membership = []
    try:
        thisUser = {}
        thisUser['USERNAME'] = user.fullName
        thisUser['EMAIL'] = user.email
        
        try: # Group membership outside the organisation will raise an error ("You do not have permissions to access this resource or perform this operation.")
            for group in user.groups:
                membership.append(group.title)
        except:
            pass
        for title in my_group_titles:
            if title in membership:
                thisUser[title] = 1
            else:
                thisUser[title] = 0
    except:
        print("PLEASE NOTE: no information can be retrieved about user "+user.fullName+".")
        pass
    writer.writerow(thisUser)
outfile.close()
print ("Ready: "+datetime.datetime.today().strftime('%c'))
print()
print('===================')
print('The CSV file can be found here:')
print(os.path.abspath(fname))
print('===================')

And here is the provide_credentials script which is used above to login to AGOL:

import json, os
from getpass import getpass #to accept passwords in an interactive fashion

def provide_credentials():
    file_with_credentials = 'my_credentials.json'
    username = ''
    password = ''
    if os.path.exists(file_with_credentials):
        with open(file_with_credentials) as f:
            data = json.load(f)
            username = data['username']
            password = data['password']
    if not username or username == 'USERNAME' or not password or password == 'PASSWORD':
        username = input('Please enter your username: ')
        password = getpass('Please enter your password (this will remain invisible): ')
    return username, password

With the input file my_credentials.json:

{
	"username":"USERNAME",
	"password":"PASSWORD"
}

Happy coding!

Egge-Jan

10 Comments
JeffTimm
Occasional Contributor

Forgive my ignorance but where in the world would 

print(os.path.abspath(fname))

be if I ran this on and Enterprise Notebook?

I ran this with no errors and it said:

=================== The CSV file can be found here: /arcgis/AGOL_Group_Membership20200416.csv ===================

But I cannot find this file
Egge-Jan_Pollé
MVP Regular Contributor

Hi Jeff Timm,

Has the file not just been created in the same folder where the Python script resides?

BR,

Egge-Jan

Katie_Clark
MVP Regular Contributor

Egge-Jan Pollé‌, this seems extremely useful, and I really appreciate you sharing! This is the type of more advanced GIS work (utilizing the API's) that I want to get in to, but for someone who is a complete newbie, I'm a bit unclear where to start. I've written/modified scripts to use in geoprocessing, but never involving multiple python files or interacting with my Organization account. Do you think you could go into a bit more detail how to approach this? (also, does this script work with both AGOL and Enterprise accounts?)

I created the JSON file and two separate .PY scripts from the code you provided, all in the same folder. Then I added the "agol_group_membership.py" script to a toolbox in ArcGIS Pro and attempted to run the tool. This is the error message I got:

Obviously I'm just not understanding how to set it up correctly. I know it's a lot to ask, but if you could at least maybe share a resource that helps explain how to get started with this stuff that would be SO helpful! 

Wishing you all the best,

Katherine

Egge-Jan_Pollé
MVP Regular Contributor

Hi Katherine Clark,

Please have a look at the screen capture of the DOSBox below.

  1. I did create a new folder on my C:\-drive specifically for this test (Yes, the folder is called Katherine :-))
  2. In this folder I placed the two files agol_group_membership.py and provide_credentials.py (for the content of these files, see code in post above)
  3. And then I did run the following command: "C:\Program Files\ArcGIS\Pro\bin\Python\scripts\propy.bat" agol_group_membership.py

Please note: It is very important to run this script in combination with the bat file mentioned to make sure you run it in the correct ArcGIS/Python environment (This batch file comes with the installation of ArcGIS Pro). Without making sure you use this correct environment the script will fail anyway...

Please let me know if this works for you.

Stay safe, stay home - have a nice weekend.

Egge-Jan

Katie_Clark
MVP Regular Contributor

Thank you! I got it to run once I changed the credentials in the JSON to be my ArcGIS Online login rather than my Enterprise login (is there a way to make this work with Enterprise?)

Strangely, I did receive these messages about not having permissions:

However, there was a csv produced in the same folder where the scripts are stored, as expected. The only bad part....the csv has different rows for each user, but no column separations. Have you experienced this before? 

Katie_Clark
MVP Regular Contributor

Update: I was able to correct the csv file by using the "Text to Columns" tool in the Data tab of the Excel Ribbon since all the values were neatly separated by semi colons.

However, still curious if this is expected behavior for the output of the script. 

by Anonymous User
Not applicable

Dumb question. I wanted to find documentation about this:

"gis.users.search(max_count=...)"

Where would I find it? The answer will be embarrassing, but I can't find it. I've searched the ArcGIS API for python documentation and I don't see an explanation of this property. 

I was getting a list of users and did not realize there was a limit of 100 returned and was getting extremely frustrated with the results, wondering why they were incomplete. I found your post and discovered the "max_count" property, and am now wondering how I could go straight to the source and find such information more directly.

Thank you,

Randy McGregor

ErikLash1
Occasional Contributor

Easiest way to get it to get it to automatically open in Excel as a table is to change the delimiter to a comma from a semi colon.  Since we can use whatever delimeters we wish the decision on what to include here is based on the needs of the client application that will be consuming the output. 

ErikLash1
Occasional Contributor

Not sure if anyone answered your question regarding making it work with Enterprise.

The answer is yes it will work with any 'portal' of whether that portal is enterprise on premesis, cloud, or arcgisonline. 

Just enter the details of the portal under the login comment at the top.

# Log in
username, password = provide_credentials()
my_agol = GIS("https://yourEnterprise.com/PortalWebadaptor", username, password)

If the variable name "my_agol" seems a bit like a mental roadblock, you can rename it throughout the script to something like "my_portal". 

Egge-Jan shared a highly adaptable and fantastically useful nugget of code here that makes short work of pulling info into spreadsheets about users and groups.

If like me you want the code to work on all your portals without much modification there are a few things you can do such as automatically naming your output csv by pulling the portal name directly via implementation of "my_portal.properties.portalName" in the fname= variable.

DataOfficer
Occasional Contributor III

Could anyone suggest how this could be modified to list users within the Groups, where those users are not from the same organisation? I can get it to list our organisation's users and their Group memberships, but it excludes the hundreds of users we have in those Groups within our hub community organisation.

Many thanks.

About the Author
GIS Consultant at Avineon-Tensing