Do you recognize this?
- "Over time our ArcGIS Online organization has grown. We have quite a few users now and a considerable number of groups has been created. Can you please provide me with a quick overview of who is member of which group?"
- "Yes, I can."
- "OK - and please note: not only quick, but also in Excel. Is that possible?"
- "Eh... yeah, that is also possible."
We are going to use the ArcGIS API for Python with groups.search() and users.search() to get an overview of our organization. Next we compile a matrix - users along the y-axis and groups along the x-axis - with a 1 or a 0 to indicate wether the user is a member of the group or not. This matrix is being written to a CSV file.
And then the boss should be able to import this CSV into a colorful and manageable spreadsheet. With or without a little help.
Does the script below work for you? Just 'like' or 'share' if it does. And if you want to, you can just open your Jupyter Notebook to try it line by line.
<SPAN class="comment token">## ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</SPAN>
<SPAN class="comment token">## ArcGIS Online Management Information</SPAN>
<SPAN class="comment token">## Script: agol_group_membership.py</SPAN>
<SPAN class="comment token">## Goal: to create an overview of group membership in your ArcGIS Online organisation</SPAN>
<SPAN class="comment token">## Author: Egge-Jan Polle - Tensing GIS Consultancy</SPAN>
<SPAN class="comment token">## Date: August 3, 2018</SPAN>
<SPAN class="comment token">## ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</SPAN>
<SPAN class="comment token">#</SPAN>
<SPAN class="comment token"># This script should be run within a specific ArcGIS/Python environment using the batch file below</SPAN>
<SPAN class="comment token"># (This batch file comes with the installation of ArcGIS Pro)</SPAN>
<SPAN class="comment token"># "C:\Program Files\ArcGIS\Pro\bin\Python\scripts\propy.bat" agol_group_membership.py</SPAN>
<SPAN class="comment token">#</SPAN>
<SPAN class="keyword token">import</SPAN> csv<SPAN class="punctuation token">,</SPAN>os<SPAN class="punctuation token">,</SPAN> sys
<SPAN class="keyword token">from</SPAN> arcgis<SPAN class="punctuation token">.</SPAN>gis <SPAN class="keyword token">import</SPAN> GIS
<SPAN class="keyword token">from</SPAN> provide_credentials <SPAN class="keyword token">import</SPAN> provide_credentials
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'==================='</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'The script that is running: '</SPAN> <SPAN class="operator token">+</SPAN> __file__<SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'First you have to log in to ArcGIS Online'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token"># Log in</SPAN>
username<SPAN class="punctuation token">,</SPAN> password <SPAN class="operator token">=</SPAN> provide_credentials<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
my_agol <SPAN class="operator token">=</SPAN> GIS<SPAN class="punctuation token">(</SPAN><SPAN class="string token">"https://www.arcgis.com"</SPAN><SPAN class="punctuation token">,</SPAN> username<SPAN class="punctuation token">,</SPAN> password<SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">"Start: "</SPAN><SPAN class="operator token">+</SPAN>datetime<SPAN class="punctuation token">.</SPAN>datetime<SPAN class="punctuation token">.</SPAN>today<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>strftime<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'%c'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">## Get all groups</SPAN>
my_groups <SPAN class="operator token">=</SPAN> my_agol<SPAN class="punctuation token">.</SPAN>groups<SPAN class="punctuation token">.</SPAN>search<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">## Optionally: have a look at all groups</SPAN>
<SPAN class="comment token">#my_groups</SPAN>
<SPAN class="comment token">## Optionally: count the number of groups</SPAN>
<SPAN class="comment token">#len(my_groups)</SPAN>
<SPAN class="comment token">## Get all users</SPAN>
my_users <SPAN class="operator token">=</SPAN> my_agol<SPAN class="punctuation token">.</SPAN>users<SPAN class="punctuation token">.</SPAN>search<SPAN class="punctuation token">(</SPAN>max_users <SPAN class="operator token">=</SPAN> <SPAN class="number token">350</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="comment token"># The default of max_users = 100, so increase it if you have more</SPAN>
<SPAN class="comment token">## Optionally: have a look at all users</SPAN>
<SPAN class="comment token">#my_users</SPAN>
<SPAN class="comment token">## Optionally: count the number of users</SPAN>
<SPAN class="comment token">#len(my_users)</SPAN>
<SPAN class="comment token">## Create a list with all group titles</SPAN>
my_group_titles <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="keyword token">for</SPAN> my_group <SPAN class="keyword token">in</SPAN> my_groups<SPAN class="punctuation token">:</SPAN>
my_group_titles<SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN>my_group<SPAN class="punctuation token">.</SPAN>title<SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">## Create a list with field names</SPAN>
fieldnames <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN>
fieldnames <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="string token">'USERNAME'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'EMAIL'</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="keyword token">for</SPAN> title <SPAN class="keyword token">in</SPAN> my_group_titles<SPAN class="punctuation token">:</SPAN>
fieldnames<SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN>title<SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">## Optionally: have a look at the field names</SPAN>
<SPAN class="comment token">#fieldnames</SPAN>
<SPAN class="comment token">## Create a CSV file with a matrix of the groups with their members</SPAN>
today <SPAN class="operator token">=</SPAN> datetime<SPAN class="punctuation token">.</SPAN>datetime<SPAN class="punctuation token">.</SPAN>today<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>strftime<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'%Y%m%d'</SPAN><SPAN class="punctuation token">)</SPAN>
fname <SPAN class="operator token">=</SPAN> <SPAN class="string token">'AGOL_Group_Membership'</SPAN><SPAN class="operator token">+</SPAN>today<SPAN class="operator token">+</SPAN><SPAN class="string token">'.csv'</SPAN>
<SPAN class="keyword token">try</SPAN><SPAN class="punctuation token">:</SPAN>
os<SPAN class="punctuation token">.</SPAN>remove<SPAN class="punctuation token">(</SPAN>fname<SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">except</SPAN> OSError<SPAN class="punctuation token">:</SPAN>
<SPAN class="keyword token">pass</SPAN>
outfile <SPAN class="operator token">=</SPAN> open<SPAN class="punctuation token">(</SPAN>fname<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'a'</SPAN><SPAN class="punctuation token">)</SPAN>
writer <SPAN class="operator token">=</SPAN> csv<SPAN class="punctuation token">.</SPAN>DictWriter<SPAN class="punctuation token">(</SPAN>outfile<SPAN class="punctuation token">,</SPAN> delimiter <SPAN class="operator token">=</SPAN> <SPAN class="string token">';'</SPAN><SPAN class="punctuation token">,</SPAN> lineterminator<SPAN class="operator token">=</SPAN><SPAN class="string token">'\n'</SPAN><SPAN class="punctuation token">,</SPAN> fieldnames<SPAN class="operator token">=</SPAN>fieldnames<SPAN class="punctuation token">)</SPAN>
writer<SPAN class="punctuation token">.</SPAN>writeheader<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="comment token">## Add for each user the full name and email and for each group a 1 or 0, depending on group membership</SPAN>
<SPAN class="keyword token">for</SPAN> user <SPAN class="keyword token">in</SPAN> my_users<SPAN class="punctuation token">:</SPAN>
membership <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="keyword token">try</SPAN><SPAN class="punctuation token">:</SPAN>
thisUser <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">{</SPAN><SPAN class="punctuation token">}</SPAN>
thisUser<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'USERNAME'</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> user<SPAN class="punctuation token">.</SPAN>fullName
thisUser<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'EMAIL'</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> user<SPAN class="punctuation token">.</SPAN>email
<SPAN class="keyword token">try</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="comment token"># Group membership outside the organisation will raise an error ("You do not have permissions to access this resource or perform this operation.")</SPAN>
<SPAN class="keyword token">for</SPAN> group <SPAN class="keyword token">in</SPAN> user<SPAN class="punctuation token">.</SPAN>groups<SPAN class="punctuation token">:</SPAN>
membership<SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN>group<SPAN class="punctuation token">.</SPAN>title<SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">except</SPAN><SPAN class="punctuation token">:</SPAN>
<SPAN class="keyword token">pass</SPAN>
<SPAN class="keyword token">for</SPAN> title <SPAN class="keyword token">in</SPAN> my_group_titles<SPAN class="punctuation token">:</SPAN>
<SPAN class="keyword token">if</SPAN> title <SPAN class="keyword token">in</SPAN> membership<SPAN class="punctuation token">:</SPAN>
thisUser<SPAN class="punctuation token">[</SPAN>title<SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> <SPAN class="number token">1</SPAN>
<SPAN class="keyword token">else</SPAN><SPAN class="punctuation token">:</SPAN>
thisUser<SPAN class="punctuation token">[</SPAN>title<SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> <SPAN class="number token">0</SPAN>
<SPAN class="keyword token">except</SPAN><SPAN class="punctuation token">:</SPAN>
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">"PLEASE NOTE: no information can be retrieved about user "</SPAN><SPAN class="operator token">+</SPAN>user<SPAN class="punctuation token">.</SPAN>fullName<SPAN class="operator token">+</SPAN><SPAN class="string token">"."</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">pass</SPAN>
writer<SPAN class="punctuation token">.</SPAN>writerow<SPAN class="punctuation token">(</SPAN>thisUser<SPAN class="punctuation token">)</SPAN>
outfile<SPAN class="punctuation token">.</SPAN>close<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">"Ready: "</SPAN><SPAN class="operator token">+</SPAN>datetime<SPAN class="punctuation token">.</SPAN>datetime<SPAN class="punctuation token">.</SPAN>today<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>strftime<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'%c'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'==================='</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'The CSV file can be found here:'</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>os<SPAN class="punctuation token">.</SPAN>path<SPAN class="punctuation token">.</SPAN>abspath<SPAN class="punctuation token">(</SPAN>fname<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="string token">'==================='</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></SPAN>And here is the provide_credentials script which is used above to login to AGOL:
<SPAN class="keyword token">import</SPAN> json<SPAN class="punctuation token">,</SPAN> os
<SPAN class="keyword token">from</SPAN> getpass <SPAN class="keyword token">import</SPAN> getpass <SPAN class="comment token">#to accept passwords in an interactive fashion</SPAN>
<SPAN class="keyword token">def</SPAN> <SPAN class="token function">provide_credentials</SPAN><SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
file_with_credentials <SPAN class="operator token">=</SPAN> <SPAN class="string token">'my_credentials.json'</SPAN>
username <SPAN class="operator token">=</SPAN> <SPAN class="string token">''</SPAN>
password <SPAN class="operator token">=</SPAN> <SPAN class="string token">''</SPAN>
<SPAN class="keyword token">if</SPAN> os<SPAN class="punctuation token">.</SPAN>path<SPAN class="punctuation token">.</SPAN>exists<SPAN class="punctuation token">(</SPAN>file_with_credentials<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
<SPAN class="keyword token">with</SPAN> open<SPAN class="punctuation token">(</SPAN>file_with_credentials<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> f<SPAN class="punctuation token">:</SPAN>
data <SPAN class="operator token">=</SPAN> json<SPAN class="punctuation token">.</SPAN>load<SPAN class="punctuation token">(</SPAN>f<SPAN class="punctuation token">)</SPAN>
username <SPAN class="operator token">=</SPAN> data<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'username'</SPAN><SPAN class="punctuation token">]</SPAN>
password <SPAN class="operator token">=</SPAN> data<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'password'</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="keyword token">if</SPAN> <SPAN class="operator token">not</SPAN> username <SPAN class="operator token">or</SPAN> username <SPAN class="operator token">==</SPAN> <SPAN class="string token">'USERNAME'</SPAN> <SPAN class="operator token">or</SPAN> <SPAN class="operator token">not</SPAN> password <SPAN class="operator token">or</SPAN> password <SPAN class="operator token">==</SPAN> <SPAN class="string token">'PASSWORD'</SPAN><SPAN class="punctuation token">:</SPAN>
username <SPAN class="operator token">=</SPAN> input<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Please enter your username: '</SPAN><SPAN class="punctuation token">)</SPAN>
password <SPAN class="operator token">=</SPAN> getpass<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'Please enter your password (this will remain invisible): '</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">return</SPAN> username<SPAN class="punctuation token">,</SPAN> password<SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></SPAN>With the input file my_credentials.json:
<SPAN class="punctuation token">{</SPAN>
<SPAN class="string token">"username"</SPAN><SPAN class="punctuation token">:</SPAN><SPAN class="string token">"USERNAME"</SPAN><SPAN class="punctuation token">,</SPAN>
<SPAN class="string token">"password"</SPAN><SPAN class="punctuation token">:</SPAN><SPAN class="string token">"PASSWORD"</SPAN>
<SPAN class="punctuation token">}</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></SPAN>Happy coding!
Egge-Jan