Listing Feature Classes Used By ArcGIS Server Services

RogerDunnGIS · ‎02-11-2022

Arriving at a new organization, I wasn't sure how our geodatabase feature classes and tables were used in ArcGIS Server. If I want to shut down services locking down a feature class, which services should I shut down? If I am replacing a feature class with a new one, which map documents and Pro projects should I modify and republish? The script I created has served me, and I wanted to share it with you, after obfuscating our connection details.

As I am not allowed to attach .ipynp files to a post, there will be some copying and pasting involved. I will describe exactly what you have to do to list out all of your services' dependencies. You can then search the CSV output for the feature classes and tables you're wondering about, and see which services use them.

Open Start > ArcGIS > Jupyter Notebook
Browse to a folder where you would like to store a script about passwords, then click New > Python 3. You'll have to change the name later. Until then, it will be named Untitled.ipynp
Copy and paste the following code into the cell:

import keyring
keyring.set_password("ArcGIS Portal", "admin", "password")

Change "admin" to the username of your ArcGIS Server/Portal administrator
Change "password" to the password of that account
Run the cell by pressing Ctrl+Enter. There will be no output and that's okay
Close the browser tab representing the file
In the Jupyter window listing your folders and files, check the box next to Untitled.ipynb (its icon will be a green book because it is running)
Click the orange Shutdown button at the top of the list
Check the box next to Untitled.ipynb again
Click the Rename button at the top of the list
Rename your file. This file should be safeguarded by you. You should NOT publish this file to a blog, code-sharing web site, or put it on a file server where lay users of your organization can find it, as it is pure text and contains a powerful password. But you need to perform this step to run the code that will follow
Once again in Jupyter, browser to a folder where you want the next script and click New > Python 3
Copy and paste the code at the bottom of this post into the cell. Feel absolutely free to split the code into logical blocks. With your cursor at a desired spot in the code block, hit Ctrl+Shift+Minus. This will allow you run blocks of code, one at a time, as you debug the code for your needs
Read through the code, replacing text constants as needed. For instance, replace variables like the adminURL with the actual URL to your ArcGIS Server Administrator Directory and tokenURL with the actual URL of your token generator. [Finding these URLs is outside the scope of this post and can be found in the product documentation]
With code divided into chunks as you please, run a cell using keyboard shortcuts like Ctrl+Enter (to run and keep the cell selected), Shift+Enter (to run the cell and select the next one down), or Alt+Enter (to run the cell and insert a new cell below). Feel free to add print() statements to show the output of various commands
Scripts are meant to be modified. If a CSV output isn't what you want, modify the script to suit your needs! Perhaps you want to pump the output to a real database table, a pandas DataFrame, or one huge JSON object, the choice is yours! The tricky part is the many-to-many nature of it all (many services can use a feature class, and many feature classes can be part of one service)

# The requests library is necessary for making http(s) calls to REST services
# The keyring library is necessary for obtaining the password for your ArcGIS Server username.  See documentation below.
# The csv library is necessary because the final output for this script is a CSV file.
# The json library is necessary for pretty printing JSON output.  Humans can more easily see property names, values,
# and parent/child relationships
import requests, keyring, csv, json
# On any one system, Python programmers can import the keyring module and call set_password with a type, user, and password.
# I have done that on my system, and assigned the password via keyring so that it does not appear in this script.
# I access that password using the type and user that was originally used in the call to set_password.
keyringType = "ArcGIS Portal"
# The user name of an ArcGIS Server administrator, used in keyring to store its password
administratorUser = "admin"
# How long the ArcGIS Server token will last
tokenExpiry = 30 # In minutes
# The kind of client looking for a token
tokenClient = "referrer"
# When in Jupyter Notebook, this is the base URL host in the address bar, and is used in obtaining a token
tokenReferer = "localhost:8888"
# The URL to generate a token in ArcGIS Server, or, if using Federated servers, ArcGIS Portal.
tokenURL = "https://maps.yourentity.com/portal/sharing/rest/generateToken"
# The ArcGIS Server Admin REST URL, tied to a web adaptor.  Sometimes this has a :6443 in it.
adminURL = "https://maps.yourentity.com/server/admin"
servicesURL = adminURL + "/services"
# The name of the output CSV file for this script
outputCSVName = "ArcGISServerDump.csv"
# The format of the output CSV file.  See https://docs.python.org/3/library/csv.html#dialects-and-formatting-parameters for more info.
csvDialect = "excel"
# Do not change this constant.  It is appended to URLs to obtain the service manifest, which contains a lot of important
# metadata about the service.
manifestSuffix = "iteminfo/manifest/manifest.json"
# Do not change this constant.  These are URL parameters for obtaining what this script expects
servicesParams = {"detail": "false", "f": "json"}
# Do not change this constant.  This is a URL parameter for obtaining what this script expects
jsonParams = {"f": "json"}
# Obtain the password stored in keyring
password = keyring.get_password(keyringType, administratorUser)
# Do not change this variable.  These are URL parameters sent in the POST to obtain a token
tokenParams = {"username": administratorUser, "password": password, "expiration": tokenExpiry, "client": tokenClient, "referer": tokenReferer, "f": "json"}
# Obtain an ArcGIS Server token
r = requests.post(tokenURL, data=tokenParams)
# Show the http status code
print("Http Status Code Returned:", r.status_code)
# Transform the result to JSON
result = r.json()
# Get the token
token = result["token"]
# Show the token, so it can be copied and pasted, if needed, for debugging, or using in a browser
print(token)
# Do not change this variable.  These are http headers for obtaining responses using authentication and authorization
# via ArcGIS Server and/or Portal
servicesHeader = {"Content-Type": "application/json", "X-Esri-Authorization": "Bearer {0}".format(token)}
# Get all of the services in an http response
servicesResponse = requests.get(servicesURL, params=servicesParams, headers=servicesHeader)
# Transform the response to JSON
servicesJson = servicesResponse.json()
# Get a list of all the folders
gisFolders = servicesJson["folders"]
# Since the root folder is never listed, add it at the front of a new list
realFolders = ["Root"] + gisFolders
# Optionally, print the list of ArcGIS Server folders
print(realFolders)
# Create/overwrite the output CSV.  If the CSV is open, especially in Excel, then this part of the script will fail
with open(outputCSVName, "w", newline='') as outFile:
    # Create a CSV Dictionary Writer object
    outWriter = csv.DictWriter(outFile, quoting=csv.QUOTE_MINIMAL, dialect="excel"
            , fieldnames=["Folder", "Service", "Type", "Manifest", "Source#", "Source", "Client", "Database#", "Server", "Database", "Version", "Username", "Layer#", "Table"])
    # Write the names of the fields as the first line in the file
    outWriter.writeheader()
    # For every ArcGIS Server folder...
    for folder in realFolders:
        # Create (or replace) an empty output dictionary (outionary)
        outionary = {}
        # Construct the service URL differently for the root folder than for other folders
        outionary["Folder"] = folder
        if folder == "Root":
            servicesListURL = servicesURL
        else:
            servicesListURL = "{0}/{1}".format(servicesURL, folder)
        # Get an http response of the services in the current folder
        servicesResponse = requests.get(servicesListURL, params=servicesParams, headers=servicesHeader)
        # Transform the response to JSON
        servicesJson = servicesResponse.json()
        # Get a JSON list of the services
        gisServices = servicesJson["services"]
        # For every service in the folder...
        for gisService in gisServices:
            # Wipe out dictionary values assigned in other parts of this loop.  I don't use the del command because the dictionary
            # might not have these keys yet.
            outionary["Source#"] = outionary["Source"] = outionary["Client"] = outionary["Database#"] = outionary["Layer#"] = outionary["Table"] = ""
            # Assign the Service and Type in the dictionary, which map to the CSV DictWriter fields above
            outionary["Service"] = gisService["serviceName"]
            outionary["Type"] = gisService["type"]
            # This script is only interested in MapServers and ImageServers.  You can change this if statement to query other types of services
            if gisService["type"] == "MapServer" or gisService["type"] == "ImageServer":
                # Construct the URL to the manifest file
                # Construct the URL to the manifest file
                if folder == "Root":
                    manifestURL = "{0}/services/{1}.{2}/{3}".format(adminURL, gisService["serviceName"], gisService["type"], manifestSuffix)
                else:
                    manifestURL = "{0}/services/{1}/{2}.{3}/{4}".format(adminURL, folder, gisService["serviceName"], gisService["type"], manifestSuffix)
                # Assign the Manifest value in the dicionary, which will be written out later
                outionary["Manifest"] = manifestURL
                # Obtain an http response of the manifest file
                manifestResponse = requests.get(manifestURL, params=jsonParams, headers=servicesHeader)
                # Http status 200 is a good thing
                if manifestResponse.status_code == 200:
                    # Transform the response to JSON
                    manifestJson = manifestResponse.json()
                    # If a property called databases is in the response...
                    if "databases" in manifestJson:
                        # Start counting
                        iBase = 0
                        # For every database listed in databases...
                        for database in manifestJson["databases"]:
                            # Assign the Database# index number in the dictionary
                            outionary["Database#"] = iBase
                            outWriter.writerow(outionary)
                            iBase = iBase + 1
                            # If the database has an onServerConnectionString value, read it.  Other possible, meaningful values are
                            # onClientConnectionString.
                            if "onServerConnectionString" in database:
                                # The connection string looks like most other database connection strings you've seen.
                                connectionString = database["onServerConnectionString"]
                                # The connection string is a list of name=value pairs, separated by semicolons, at least in my experience,
                                # when dealing with geodatabases.  But onServerConnectionString can also be a file path.
                                # Create an empty dictionary for connection parameters
                                connectDict = {}
                                if ";" in connectionString:
                                    connectStringParams = connectionString.split(";")
                                    # For each connection parameter in the connection string...
                                    for connectParam in connectStringParams:
                                        # Wipe out existing values in the output dictionary, from Server to Username
                                        outionary["Server"] = outionary["Database"] = outionary["Version"] = outionary["Username"] = ""
                                        # Split this connection parameter into name=value
                                        nameValue = connectParam.split("=")
                                        # For example, 'USER=sde' will result in a dictionary entry "USER": "sde" in connectDict.
                                        connectDict[nameValue[0]] = nameValue[1]
                                        # Search for specific names in the connection parameters and write them to the output dictionary.
                                        # Your database connection strings may have different values that those shown here.  If this is
                                        # the case, modify these if and then statements to reflect name=value pairs in those.
                                        if "SERVER" in connectDict:
                                            outionary["Server"] = connectDict["SERVER"]
                                        if "DATABASE" in connectDict:
                                            outionary["Database"] = connectDict["DATABASE"]
                                        if "VERSION" in connectDict:
                                            outionary["Version"] = connectDict["VERSION"]
                                        if "USER" in connectDict:
                                            outionary["Username"] = connectDict["USER"]
                                else:
                                    # See above for similar logic
                                    nameValue = connectionString.split("=")
                                    connectDict[nameValue[0]] = nameValue[1]
                                    if "DATABASE" in connectDict:
                                        outionary["Database"] = connectDict["DATABASE"]
                            # Start counting layers and tables that come from that database
                            iLayer = 0
                            # For each dataset listed...
                            for dataset in database["datasets"]:
                                # Add the layer's index to the output dictionary
                                outionary["Layer#"] = iLayer
                                iLayer = iLayer + 1
                                # Add the layer's feature class or table name to the output dictionary
                                outionary["Table"] = dataset["onServerName"]
                                # Write the row to the output.  This will result in multiple rows containing similar information,
                                # but the redundancy will be helpful
                                outWriter.writerow(outionary)
                    # Now that we're done with the databases node, wipe out database properties in the output dictionary
                    outionary["Server"] = outionary["Database"] = outionary["Version"] = outionary["Username"] = outionary["Layer#"] = outionary["Table"] = ""
                    # If there is a resources section in the response...
                    if "resources" in manifestJson:
                        # Start counting resources
                        iResource = 0
                        # For each resource list...
                        for resource in manifestJson["resources"]:
                            # Add the resource's index to the output dictionary
                            outionary["Source#"] = iResource
                            iResource = iResource + 1
                            # If an onPremisePath is listed...
                            if "onPremisePath" in resource:
                                # Write the path to the output dictionary, which is typically the name of the ArcMap document or ArcGIS Pro project
                                # which published the service.
                                outionary["Source"] = resource["onPremisePath"]
                            if "clientName" in resource:
                                outionary["Client"] = resource["clientName"]
                            outWriter.writerow(outionary)
outfile.close()

The resulting file will likely be stored in the same place as the script. You control the name of the file with the constant outputCSVName in the script.

I hope this post has helped you, even if it has only inspired you to write your own. Feel free to follow me, as I tend to post reusable scripts that help me do my job, and they may help you use yours. And please ask questions if something fails; if I don't know the answer I'll say as much.

"Roj"

RogerDunnGIS · ‎02-11-2022

Oh, also remember to save and rename your script afterwards, and to cleanly close down the Jupyter server (DOS window) with Ctrl+C.

MellissaLasslo · ‎02-24-2022

This was very helpful for us to go through our services to be able to clean up services that are no longer in use and from some projects that were done in 2016. It was great to be able to see the sources of the data as well. Great work!!

DirtDogRoj · ‎05-25-2022

I have rewritten this script. Rather than spitting out a CSV, this new script will report all the services used by a target feature class and all of its siblings in the feature dataset. And it will list the map documents or Pro projects that directly contain the target feature class. Each code block below represents a different cell in my Python notebook, be it in Pro or Jupyter.

Like the previous script, I have imported keyring, and in a different script, I have called keyring.set_password(keyringType, administratorUser, myPassword)

# The requests library is necessary for making http(s) calls to REST services
# The keyring library is necessary for obtaining the password for your ArcGIS Server username.  See documentation below.
# The csv library is necessary because the final output for this script is a CSV file.
# The json library is necessary for pretty printing JSON output.  Humans can more easily see property names, values,
# and parent/child relationships
import requests, keyring, json, arcpy

# On any one system, Python programmers can import the keyring module and call set_password with a type, user, and password.
# I have done that on my system, and assigned the password via keyring so that it does not appear in this script.
# I access that password using the type and user that was originally used in the call to set_password.
keyringType = "ArcGIS Portal"
# The user name of an ArcGIS Server administrator, used in keyring to store its password
administratorUser = "adminUserName"
# How long the ArcGIS Server token will last
tokenExpiry = 90 # In minutes
# The kind of client looking for a token
tokenClient = "referrer"
# When in Jupyter Notebook, this is the base URL host in the address bar, and is used in obtaining a token
tokenReferer = "localhost:8888"
# The URL to generate a token in ArcGIS Server, or, if using Federated servers, ArcGIS Portal.
# Do not include a trailing slash
tokenURL = "https://www.yourcompany.com/portal/sharing/rest/generateToken"
# The ArcGIS Server Admin REST URL, tied to a web adaptor.  When you go to this URL in a browser, you should be
# presented with a salmon-and-white login dialog box titled "ArcGIS Server Administrator Directory".
# Do not include a trailing slash
adminURL = "https://www.yourcompany.com/server/admin"
# The ArcGIS REST Services Directory.  This is usually a public-facing URL with connections to your services.
# When you go to this URL, you should be presented with a white page of links to Folders and Services.
# Do not include a trailing slash
publicURL = "https://www.yourcompany.com/server/rest/services"
# The name of the geodatabase connection file.
gdbConnection = "username@database.sde"

# Obtain the password stored in keyring
password = keyring.get_password(keyringType, administratorUser)

def sisterFeatureClasses(geodatabase, featureClassName):
    walk = arcpy.da.Walk(geodatabase)
    for sdeFile, datasets, tables in walk:
        if featureClassName in tables:
            siblings = tables[:]
            siblings.remove(featureClassName)
            siblings.sort(key=str.lower)
            return siblings
    return []

# getToken returns a long token string that will authenticate you based on the parameters.
# For these purposes, it's to an ArcGIS Server administrative point.
def getToken(userName, password, tokenURL, tokenExpiry, tokenClient, tokenReferer):
    # Do not change this variable.  These are URL parameters sent in the POST to obtain a token
    tokenParams = {"username": userName, "password": password, "expiration": tokenExpiry, "client": tokenClient, "referer": tokenReferer, "f": "json"}
    # Obtain an ArcGIS Server token
    r = requests.post(tokenURL, data=tokenParams)
    if r.status_code == requests.codes.ok:
        # Transform the result to JSON
        result = r.json()
        # Get the token
        return result["token"]
    else:
        return ""

token = getToken(administratorUser, password, tokenURL, tokenExpiry, tokenClient, tokenReferer)
# Show the token, so it can be copied and pasted, if needed, for debugging, or using in a browser
print(token)

# Given the address of the Administrative Directory, and a security access token, return a list of
# folders, including the root folder which is kinda blank.
def listServicesFolders(adminURL, securityToken):
    servicesURL = adminURL + "/services"
    # Do not change this constant.  These are URL parameters for obtaining what this script expects
    servicesParams = {"detail": "false", "f": "json"}
    # Do not change this variable.  These are http headers for obtaining responses using authentication and authorization
    # via ArcGIS Server and/or Portal
    servicesHeader = {"Content-Type": "application/json", "X-Esri-Authorization": "Bearer {0}".format(securityToken)}
    # Get all of the services in an http response
    servicesResponse = requests.get(servicesURL, params=servicesParams, headers=servicesHeader)
    if servicesResponse.status_code == requests.codes.ok:
        # Transform the response to JSON
        servicesJson = servicesResponse.json()
        # Get a list of all the folders
        gisFolders = servicesJson["folders"]
        # System is a built-in folder that contains services like CachingTools and PublishingTools.  This isn't
        # a folder that contains stuff you've published.
        gisFolders.remove("System")
        # Utilities is a built-in folder that contains services like GeocodingTools and PrintingTools.  This isn't
        # a folder that contains stuff you've published.
        gisFolders.remove("Utilities")
        # Since the root folder is never listed
        gisFolders.append("")
        gisFolders.sort(key=str.lower)
        return gisFolders
    else:
        return []

# Returns a list of services in a folder.  If querying the root folder, pass "" for folder.
# The resulting list is a list of objects with the following properites: folderName, serviceName,
# type, and description.
def listServicesInFolder(adminURL, folder, securityToken):
    servicesURL = adminURL + "/services"
    # Do not change this constant.  These are URL parameters for obtaining what this script expects
    servicesParams = {"detail": "false", "f": "json"}
    # Do not change this variable.  These are http headers for obtaining responses using authentication and authorization
    # via ArcGIS Server and/or Portal
    servicesHeader = {"Content-Type": "application/json", "X-Esri-Authorization": "Bearer {0}".format(securityToken)}
    if folder:
        servicesListURL = "{0}/{1}".format(servicesURL, folder)
    else:
        servicesListURL = servicesURL
    # Get an http response of the services in the current folder
    servicesResponse = requests.get(servicesListURL, params=servicesParams, headers=servicesHeader)
    if servicesResponse.status_code == requests.codes.ok:
        # Transform the response to JSON
        servicesJson = servicesResponse.json()
        # Get a JSON list of the services
        return servicesJson["services"]
    else:
        return []

# Returns a dictionary of documents to republish without the layer, and services that need to be restarted
# if affectedFeatureClassName is modified or deleted.
# Return value type dictionary of lists {"restart": [], "republish": []}
def findServicesToRepublishAndOrRestart(adminURL, publicURL, securityToken, geodatabaseConnection, affectedFeatureClassName):
    # This will be the return type: a dictionary with two entries, restart and republish, the results
    # of which are lists of service names
    actions = {"restart": [], "republish": []}
    
    # Do not change this variable.  These are http headers for obtaining responses using authentication and authorization
    # via ArcGIS Server and/or Portal
    servicesHeader = {"Content-Type": "application/json", "X-Esri-Authorization": "Bearer {0}".format(securityToken)}
    # Do not change this constant.  It is appended to URLs to obtain the service manifest, which 
    # contains a lot of important metadata about the service.
    manifestSuffix = "iteminfo/manifest/manifest.json"
    # Do not change this constant.  This is a URL parameter for obtaining what this script expects
    jsonParams = {"f": "json"}

    featureClassToChange = affectedFeatureClassName
    classToLookFor = featureClassToChange.split(".")[2]
    featureClassesInSameDataset = sisterFeatureClasses(geodatabaseConnection, featureClassToChange)
    classesToLookFor = [x.split(".")[2] for x in featureClassesInSameDataset]
    
    servicesFolders = listServicesFolders(adminURL, securityToken)
    for folder in servicesFolders:
        gisServices = listServicesInFolder(adminURL, folder, securityToken)
        # For every service in the folder...
        for gisService in gisServices:
            if gisService["type"] in ["MapServer", "ImageServer", "FeatureServer"]:
                # Construct the URL to the manifest file
                if folder:
                    thisServiceURL = "{0}/{1}/{2}".format(publicURL, folder, gisService["serviceName"])
                    manifestURL = "{0}/services/{1}/{2}.{3}/{4}".format(adminURL, folder, gisService["serviceName"], gisService["type"], manifestSuffix)
                else:
                    thisServiceURL = "{0}/{1}".format(publicURL, gisService["serviceName"])
                    manifestURL = "{0}/services/{1}.{2}/{3}".format(adminURL, gisService["serviceName"], gisService["type"], manifestSuffix)
                # Obtain an http response of the manifest file
                manifestResponse = requests.get(manifestURL, params=jsonParams, headers=servicesHeader)
                # Http status 200 is a good thing
                if manifestResponse.status_code == requests.codes.ok:
                    # Transform the response to JSON
                    manifestJson = manifestResponse.json()
                    # foundFC means "found a service referencing the feature class"
                    foundFC = False
                    # foundDS means "found a service referencing a feature class in the same dataset"
                    foundDS = False
                    # If a property called databases is in the response...
                    if "databases" in manifestJson:
                        # For every database listed in databases...
                        for database in manifestJson["databases"]:
                            # If the database has a propery called datasets...
                            if "datasets" in database:
                                # For each dataset listed...
                                for dataset in database["datasets"]:
                                    if "onServerName" in dataset:
                                        # Add the layer's feature class or table name to the output dictionary
                                        featureClass = dataset["onServerName"]
                                        if featureClass == classToLookFor:
                                            foundFC = True
                                        if featureClass in classesToLookFor:
                                            foundDS = True
                    # If there is a resources section in the response...
                    if ("resources" in manifestJson) and foundFC:
                        # For each resource list...
                        for resource in manifestJson["resources"]:
                            # If an onPremisePath is listed...
                            if "onPremisePath" in resource:
                                # Write the path to the output dictionary, which is typically the name of the ArcMap document or ArcGIS Pro project
                                # which published the service.
                                source = resource["onPremisePath"]
                                actions["republish"].append(source)
                    if foundDS or foundFC:
                        actions["restart"].append(thisServiceURL)
    return actions

featureClassToChange = "Database.Owner.FeatureClassName"
myResult = findServicesToRepublishAndOrRestart(adminURL, publicURL, token, gdbConnection, featureClassToChange)
for action in myResult.keys():
    print(action)
    for service in myResult[action]:
        print(service)

By breaking up the code into testable functions, it is more easily tested at each step. I hope this proves useful to those who find this. Also, I apologize for not providing proper doc strings for each function. I wish to improve, not in documentation (because I'm good at that), but in documenting things the way Python would rather they be documented. Thanks for your support!

ShawnBeecher · ‎03-31-2023

This has been extremely helpful

StellaSpring · ‎05-25-2023

Roger,
Thanks so much for your work and for posting these scripts. This saved a whole lot of time.