Using Jupyter Notebooks to search for all items that contain a specific feature service?

[DaveK]

Hello, 

I'm trying to create a Python script within Jupyter Notebooks that will allow me to search all publicly available items within ArcGIS online that contain a specific feature service. Here is the script I have so far. 

import arcgis
from arcgis.gis import GIS

# connect to ArcGIS Online
gis = GIS("https://www.arcgis.com")
feature_service_id = "FEATURE SERVICE ID"
print('connected to arcgis online')

# search for all publicly available feature services
web_maps = gis.content.search(query= feature_service_id, item_type = 'web map', outside_org=True, max_items = 200)
print('Found public items')

# print the titles of the feature services
for item in web_maps:
    print('NAME: ' + item.title,'OWNER: ' + item.owner)
print('completed')

I cant seem to figure out how to return the webmaps that contain the feature service specified. Any help is appreciated. 

Thanks.

[jcarlson]

To find instances of the feature service, you need to look at the JSON definition of the maps and look for a matching URL. You might also want to look through public apps, as a web app can reference the layer through a widget, and this would not show up in the map configuration.

Here's a notebook I wrote up for this exact kind of thing!

https://kendall.maps.arcgis.com/home/notebook/notebook.html?id=2967bae3767640038a6841714784d44c

[PeterKnoop]

Great script @jcarlson ! There are a few additional, common use cases one might want to check on too:

• Insights Workbooks (which you should be able to add to the list of Web Apps in your script.)
• Views, Service Definitions, Exports, etc. related to the feature service, which can be retrieved with related_items(). 
• Survey123 Forms and their feature service, which can be retrieved with get_data() and in the zip looking for the feature service reference in the form.info file.

It isn't really feasible for a brute force scripting approach to identify all the possible dependencies, however, it can cover the most common use cases fairly well. What will get overlooked are edge cases like a Notebook or an Arcade Expression in a Web Map or Dashboard, where a feature service reference is generated programmatically, rather than coded into searchable text with an explicit item ID or feature service URL.

[jcarlson]

Great suggestions! Arcade should still be caught, though, as the expression text gets saved into the JSON of the map or dashboard. A layer used by Arcade is either part of the same service ($datastore or relationship class), another layer in the map ($map), or else it's brought in via the FeatureSetByPortalItem function, in which the service itemID is a parameter, so it's searchable. Probably need to tweak the code to account for this, though...

[PeterKnoop]

Most of the common Arcade use cases would have an explicit item ID or feature service URL present in the code. We have some edge cases, however, where the portal URL and item ID value passed to FeatureSetByPortalItem are programmatically retrieved from another feature service for abstraction reasons, hence there is no hardcoded item ID to be searched for in the Arcade code. Trying to develop a script that covers all potential possibilities doesn't seem feasible.

[DaveK]

Hey @jcarlson., 

Thanks for supplying this notebook! After inputting a service ID and running it I did get some inaccurate results. I tried the script with an item in my content that is within a single webmap. The results did not come back with the webmap the service was in and returned other items from my organization. Is there any editing I need to do prior to running this code? 

Thanks again!

[jcarlson]

Interesting... Well, are you able to share the ItemID? I'd be curious to test this against the actual item of content.

I would suggest looking at the JSON for the "inaccurate" items. Try going to your-org.arcgis.com/sharing/rest/content/items/<itemid>/data?f=json

Search through that text to see if the itemID or URL appear. You might need a token to hit the URL directly, so you could also use the Notebook to return the actual data JSON of matching items.

I didn't initially do that because of the size of the response, but in the last code cell, we can do this instead:

dependencies = pd.concat(
    [
        pd.DataFrame([{'title':a.title, 'id':a.id, 'type':a.type, 'url':f'{gis.url}/home/item.html?id={a.id}', 'data':a.get_data()} for a in app_list]),
        pd.DataFrame([{'title':m.title, 'id':m.id, 'type':m.type, 'url':f'{gis.url}/home/item.html?id={m.id}', 'data':m.get_data()} for m in map_list])
    ]
)

dependencies.head()

Then if there's an item in the output that you consider inaccurate, call it by index:

dependencies.loc[3,'data']

This will show you the full definition of the map / app, and if your service appears in it, you'll be able to find it.

[shapirot]

@jcarlson   Can you share this notebook again?  looks like that link requires credentials to your org now

[jcarlson]

No idea what changed about AGOL's notebook sharing, but here's a Github Gist w/ the same stuff in it.

https://gist.github.com/jdcarls2/717f79ff12a4a1cb337dafe8e76c70df

[PeterKnoop]

@jcarlson , @shapirot the notebook is shared with everyone, however, the link provided is for running the Notebook. That means ArcGIS needs to check that you have to have the ability to run Notebooks on an ArcGIS Online organization, so you get directed to an authentication workflow.

The workaround is to provide a URL to the Notebook's Overview tab on its Item Details page, which doesn't require any special privileges. You also need to use www.arcgis.com, instead of your own org, kendall.maps.arcgis.com.

Try this URL to access the notebook:

https://www.arcgis.com/home/item.html?id=2967bae3767640038a6841714784d44c#overview

This permits anyone to Preview or Download the Notebook, even if they don't have permission to run Notebooks on their own ArcGIS Online org.