Report ArcGIS Enterprise/ArcGIS Online Item Dependencies

[#1]

This tool will create a report on the dependencies for a service or a web map.  This can be helpful to see what web maps/applications will be affected by making changes, or deleting, a particular item.  When creating a report for feature services, the tool will report which web maps and web applications are consuming the service.  When creating a report for a web map, the tool will report which web applications are consuming the web map.  

A CSV file is created from this tool.  Below is an example report created for 2 feature services, Cities and NYPD_Crimes:

The Cities service is found in 2 web maps, Cities Web Map and Airports Web Map.  The Airports Web Map also has an application created from it called Airports Web App.

The NYPD_Crimes service is only found in 1 web map, NYPD Crimes Web Map.

See the video below for a demonstration:

Note:  this tool was created using the ArcGIS API for Python, and can only be executed within ArcGIS Pro, not ArcMap.

Update 8/2/21:  tool now works with ArcGIS Online.  The ArcGIS Online version can take a significant amount of time.  It's dependent on the number of users you have in your ArcGIS Online organization, as the tool iterates through each Creator user type and checks their web maps/web appbuilder applications.

Update 10/19/22:  tool will report Experience Builder and Dashboard applications.  A 'sleep' parameter has been added to the ArcGIS Online version to help avoid too many requests to ArcGIS Online, resulting in the tool to fail.

Update 9/5/23:  whether you are creating a report from ArcGIS Online, or Portal for ArcGIS, a single tool can be used.  A sleep parameter is no longer required if running against ArcGIS Online.

Update 10/6/23:  can now select multiple users

[Anonymous User]

This is what we've put together.  It seems to work but could definitely use some improvements.

from arcgis.gis import GIS
from IPython.display import display

source = GIS(GIS, username, password)

content = source.content.search(query = "NOT owner: esri", max_items=5000)

dependents = []

for item in content:
dep_list = item.dependent_upon()['list']
for element in dep_list:
if element['dependencyType'] == 'id':
if element['id'] == '10fe923f15ac422f8f4d48efb394c616':
dependents.append(item)

print(dependents)

[AndresCastillo]

Beautiful work Jake Skinner‌.

Does this tool find web appbuilder app, configurable apps, etc (excluding dashboards at this time)?

How To: Find dependencies in Portal for ArcGIS using ArcGIS API for Python 

[JakeSkinner]

Andres Castillo‌ yes, that is correct.  It has not been extensively tested, so please feel free to report any issues you run into in the comments section.

[WarrenMedernach]

Hi all,

I just went to try this out but it's not displaying all of the Users in the Portal?

It seems to be limiting it to the first 100, alphabetically...  I'm not seeing anything in the code that is limiting it, but it would seem add that it's listing users from 'a' to 'r' and there are exactly 100 users in the list...

[JakeSkinner]

Warren Medernach‌ I updated the scripts validation so it will load more than the first 100 users.

[WarrenMedernach]

Works like a charm, thanks Jake!

[amcsparron2793]

This script seems to be exactly what I was (somewhat unsuccessfully) trying to create myself... but I'm running into an issue with your script. Even with only one entry for "User Content" and one user... the script is failing after ~1 min with: Exception: Too many requests. Please try again later. (Error Code: 400).

@JakeSkinner Any ideas?

[JakeSkinner]

@amcsparron2793 can you send a screen shot of how you have the tool configured?

[amcsparron2793]

@JakeSkinner I copy and pasted the log below, I couldn't figure out how to attach a text file sorry. If there's other info you need I'm happy to provide it.

Create Item Dependencies Report (AGOL)
=====================
Parameters

AGOL URL https://albanynywater.maps.arcgis.com/
AGOL Username amcsparron
Password *****
Source User amcsparron
Content Type Web Maps
User Content 'Fire Hydrants With and Without Locks - Web Map - f88e9a5aff9d425a9b0cd9f6edf37b2b'
Output CSV C:\Users\AMcSparron\Desktop\Output.csv
=====================
Messages

Start Time: Monday, January 24, 2022 1:14:47 PM
==========================
Processing Fire Hydrants With and Without Locks
Traceback (most recent call last):
File "C:\Users\AMcSparron\Desktop\Item Dependency\Item_Dependency_AGOL.py", line 124, in <module>
dependencies = checkWebApps(item.id, dependencies)
File "C:\Users\AMcSparron\Desktop\Item Dependency\Item_Dependency_AGOL.py", line 40, in checkWebApps
searchResults = gis.content.search(query=f"owner:{user}", item_type="Web Mapping Application", max_items = 10000)
File "C:\Users\AMcSparron\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\__init__.py", line 5888, in search
sort_order=sort_order,
File "C:\Users\AMcSparron\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\__init__.py", line 5681, in advanced_search
data = future.result()
File "C:\Users\AMcSparron\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\concurrent\futures\_base.py", line 428, in result
return self.__get_result()
File "C:\Users\AMcSparron\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "C:\Users\AMcSparron\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\AMcSparron\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\__init__.py", line 5635, in advanced_search
as_dict=as_dict,
File "C:\Users\AMcSparron\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\_impl\_search.py", line 142, in _search
res = gis._con.post(url, params)
File "C:\Users\AMcSparron\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 1079, in post
force_bytes=kwargs.pop("force_bytes", False),
File "C:\Users\AMcSparron\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 625, in _handle_response
self._handle_json_error(data["error"], errorcode)
File "C:\Users\AMcSparron\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 648, in _handle_json_error
raise Exception(errormessage)
Exception: Too many requests. Please try again later.
(Error Code: 400)

Failed script Create Item Dependencies Report (AGOL)...
Failed to execute (ItemDependenciesReportAGOL).
Failed at Monday, January 24, 2022 1:15:48 PM (Elapsed Time: 1 minutes 0 seconds)

[PeterKnoop]

At some point it appears that Esri changed the underlying limits on requests like this, hence, the "Too many requests" message. If you are running a lot of searches, like in this script, then you need to add a sleep timer to slow down the rate of our requests. 

We haven't figured out what the new limits are, nor have we come across any documentation, however, similar scripts that started failing for us with that error are now working fine again with a sleep inserted. Typically sleeping for 5 or 10 seconds between queries was required. Unfortunately that means your scripts will take much longer to complete.

[amcsparron2793]

@PeterKnoop makes sense, but weird that we were never informed of the limit. Thanks for the hint about sleep time, I never know whether its talking "try again in a few seconds" or "try again tomorrow" lol.

[JasonBatory1]

@PeterKnoop I am getting the same "Too many requests. Please try again later" error 400 as above when running the script. 

Are you able to please provide more information as to how you modified the script with a sleep timer and where to insert the code so that the Item Dependencies Report (AGOL) scripts can run to completion? Cheers.

[PeterKnoop]

@JasonBatory1 I don't have the code in front of me, however, I would insert a sleep() call anywhere that repeated requests to ArcGIS Online are causing the error you are observing. Most likely these are requests embedded in loops, so at the end of each iteration, add a sleep(5), to pause for 5 seconds, and if that's not enough, up it to 10 more more seconds between iterations.

[John_Spence]

@PeterKnoop , on the few scripts where I run into this specific issue, I have it do a retry (x3) and then pause for 10 seconds. If there is another round of retries, I it does a 90 second pause.

[ElizabethZizzamia1]

Hello @JakeSkinner ,

I am trying to use this tool in Pro3.0 and it is failing. I've used it before with success. I enter the first three parameters and it gives me an error after I enter the password. I am the Admin for my org and we use SSO. I tried using my username and _org name (bzizzami_urichmond) as well but no luck. Any guesses? I've attached a screenshot of my screen. Thanks for any help! Beth

 

[JakeSkinner]

@ElizabethZizzamia1 SAML user account is not supported with this tool.  If trying with a built-in account, make sure you have the security question set when you first log in with this account.

[CharlottePeters95540]

Hi,

I'm trying the "Create Item Dependencies Report (AGOL)" script from ArcGIS Pro 3.0.3 and get a 403 error after entering the URL, username, and password. I'm not the admin for the account, just have publisher role. I'm able to access my organizational AGOL account in the browser.  It's not an AGOL Portal. I don't know anything about SAML or SSO.  

Thanks, Charlotte

[JakeSkinner]

@CharlottePeters95540 you need to be an Administrator in order to run this tool.

[gis_user]

Hello @JakeSkinner ,

Thank you for making the great tool! We are now trying to use the tool, and we have a question. How can we just query all "Source Users" and "Content" instead of individual user (for example from enterprise portal)? It looks that with the 10/6 update, the tool can now select multiple users, but I am not sure how to query multiple users or all the users and their contents using the tool. Any example(s) would be really appreciated. Thank you!

[JakeSkinner]

Hi @gis_user,

You can select multiple users from the Source User parameter:

[gis_user]

Thank you for the example,  @JakeSkinner!
I just downloaded the tool from the link this morning, but I am not seeing the drop-down option in the tool.. (I will download the tool again to make sure.) (I am using Pro 3.1.3.) Oh, I think the drop-down option probably shows up after entering the information above (like URL, etc.).

If I attempt to query the info for all the users in the enterprise portal (production environment, for example), how should I go about it?
Thanks again!

[JakeSkinner]

@gis_user yes, this dropdown won't show until you fill out the parameters before.  You can select all the users, and then all the users content to create a report for all items.

[PeterKnoop]

@JakeSkinner this likely only affects a few folks, however, if your Online/Portal has more than 10,000 users, then gis.users.search(max_users=100000) seems to ignore max_users, and still only returns the first 10,000 users. That has been our experience with the ArcGIS Python API up through 2.1.0.2 (the version included with ArcGIS Pro 3.1.3.) 

Our workaround has been to use gis.users.advanced_search(), and query for users by year of account creation, one year at a time, and aggregate the results. With a warning generated if any of the individual years returns 10,000 results, as that might be a sign that we've hit the hard limit of 10,000 for search, and need a different workaround, such as further reducing the creation time range being queried on each pass.

[JakeSkinner]

@PeterKnoop thank you for this, this is good to know.

[gis_user]

Thank you again for your speedy responses,  @JakeSkinner

[ZakBaron]

Thanks! This is great! I have not tested it extensively, but the first trials worked as expected.  This will be a big help to me.

[JonEmch]

Thanks for keeping this updated Jake!

[GeobilityCo]

This is awesome! I've worked on my own version of this that populated dependencies of layers used in maps but this adds the listing of maps in apps. Love it! Thank you.

I found a little something to add/correct to help. When you use the "Web Maps" option under "Content Type", the output lists the web map item itself as a dependency. I think something like

if webMap.id != app.id:

inserted right after line 25 should remove these self-references.

Thanks again!

[JakeSkinner]

@GeobilityCo thanks for pointing out this issue.  I went ahead and updated the code so that the web map is no longer being reported as a dependency of itself.

[BertKraan1]

Thanks @JakeSkinner , this is a great tool which I have been missing for years

This one got me puzzled for a minute:

it turns out I used comma's in the  title "Afdeling, boomsoort, kiemjaar" 🙂  🙂

Bert

[JakeSkinner]

@BertKraan1 thank you for bringing this to my attention.  I went ahead and updated the script to remove commas when writing to the CSV file.  You can re-download the tool and it should work correctly now.

[Luiz_RobertoArueira_da_Silva]

Good morning!

@JakeSkinner 
Excellent tool for ArcGis Enterprise administrators!
I did a test of publishing to the Portal to configure a web application, but I came across a problem which was that the users drop-down list did not appear with the options, while on the desktop this happens.
Is there any way to change the code so that the tool can query online?

Thanks!!

[JakeSkinner]

@Luiz_RobertoArueira_da_Silva yes, this tool will also work for ArcGIS Online if that is what you are referring to.

Can you send a screen shot of the tool configured when the users drop-down is not populating?

[Luiz_RobertoArueira_da_Silva]

Bom dia!
Good morning!
I wasn't referring to accessing AGOL!
I'm referring to publishing in ArcGis Enterprise, as a geoprocess and using it in a web app.
When testing, the user list was not loaded, unlike the desktop version where the list is automatically populated.
I don't know if it's a limitation, but I looked at the code and the list is fed from a search on the Portal. I thought it should work as a published tool.
Thanks.

[JakeSkinner]

@Luiz_RobertoArueira_da_Silva I see what you're asking now.  The toolbox has a lot of validation tied to, which I don't believe carries over to a GeoProcessing Service.  

https://community.esri.com/t5/arcgis-web-appbuilder-questions/how-to-implement-custom-tool-validatio...

[Luiz_RobertoArueira_da_Silva]

A feather!
Thanks.

[ODWC_GIS]

I haven't read through half of the code, and I've already learned so much!

Thank you for posting this!!  I don't know how I could've missed it in my previous searches!

[Sibe]

@JakeSkinner Thank you for creating this tool. Really helpfull and gave great results for Enterprise. Is it possible to run this in the online portal as a tool/notebook (So it's not running locally)? I tried running locally for our online portal but it took a sigificant amount of time and after 12 hours it gave a connection error. I thought integrating it in the online portal could be a solution as the connection problems won't be an issue there. 

[JakeSkinner]

@Sibe, yes this is possible.  You will just need to specify the Variables.  Below is an example.  You can remove the portal, username, and password variables. 

• The GIS object can reference "home" to connect to Portal.  
• The items variable is a semi-colon delimited string that will require a dash between the name and id. 
• The csv_output will need to reference an ArcGIS Notebook Server workspace

[Sibe]

@JakeSkinner Thank you for your reply. Is it possible to specify all services and users at the online portal at once in order to make a full report, or does it need to be specified per item/user through the method you mentioned above?

[JakeSkinner]

@Sibe yes, this is possible.  Here is a high level overview of what you would need to do:

• Iterate through all users that are not Viewers or Editors and get all of their items that are services
• Append these items to a list (i.e. items = [])
• Update line 93 to the following:  for item in items:
  
  
• Remove lines 97-101

[SFortnam]

Does this tool work with items that are shared with a Portal-to-Portal Collaboration?

[JakeSkinner]

@SFortnam this tool will only report dependencies within it's own Org.

[Sibe]

@JakeSkinner Thank you I got it to work as a notebook in ArcGIS Online. It runs much faster on the notebook server compared to locally for the Online portal. 

Another question, do you plan to implement additional functionality to see which feature classes are used in layers, webmaps and apps? Would be very helpfull in case of a datamodel change to see which items are affacted on the portal. 

[Sibe]

Regarding the question of the additional functionality above to see which feature classes are used in layers, webmaps and apps do you think this would be a helpful addition to this tool and do you have ideas how this can be configured or plans to implement this functionality into this tool? 

[JakeSkinner]

@Sibe I currently don't have plans to implement this.  To do this, you would need to query the manifest.json or manifest.xml files located at \arcgisserver\config-store\services\<service name>\esriinfo\manifest.  These files hold information of the feature classes/tables used in the services.