topic Re: Filter Services by those which contain .MapServer to get Service Manifest in ArcGIS API for Python Questions

Filter Services by those which contain .MapServer to get Service Manifest

JaymieCroucher — Thu, 29 Sep 2022 11:04:25 GMT

Hi Everyone,

Our team are trying to create a single table which creates a link between a portal item its service URL and the feature class(s) and database it sits in. Trawling the forum has led me to look at using the service manifest as a source for this for which I've had some success however I've now encountered an issue where as I loop through each Service within a folder I hit an error if the service holds no manifest. This is because geoprocessing services (e.g. A custom data export / print service) holds no manifest and so the script fails.

I've tried a TRY function and also various way of trying to create a test case to filter our non MapServer services but with little success. This is mainly as each service in the returned list isn't string but a custom ArcGIS Server data type which is limited.

Does anyone have any ideas on how to filter services?

---------------------------------------------------------------------------------------------------------------------------------------

from arcgis.gis import *
import json

gis = GIS("portal", "username", "password",verify_cert=False,)

print("Successfully logged in as: " + gis.properties.user.username)

servers = gis.admin.serverss = servers.list() #list federated servers
server1 = s[1] #isolate portal server in list
services = server1.services.list(folder="Shared") #list the services within the folder "Shared"

for service in services: #list the manifest for each service URL
ii = service.iteminformation
manifest = ii.manifest
a = json.dumps(manifest)
b = json.loads(a)
fcnam = b['databases'][0]['datasets'] #scan the JSON and fine the list of feature classes
dbnam = b['databases'][0]['onServerName'] # scan the JSON and find the DB name
for x in fcnam:
print(service,dbnam,x['onServerName']) #print service with list of db fc names

Re: Filter Services by those which contain .MapServer to get Service Manifest

RobertAkroyd1 — Thu, 06 Oct 2022 16:16:03 GMT

Hi @JaymieCroucher

I think all you need to do what you are asking above to filter on MapServer services is to add the following inside your "for service in services", e.g.

for service in services: #list the manifest for each service URL

if service.type == "MapServer":
ii = service.iteminformation

# the rest of your code

Re: Filter Services by those which contain .MapServer to get Service Manifest

JaymieCroucher — Thu, 06 Oct 2022 18:53:18 GMT

Very grateful Rob, worked a dream 😀

Re: Filter Services by those which contain .MapServer to get Service Manifest

HaThach1 — Mon, 14 Nov 2022 20:49:18 GMT

This script print out a few mapserver and then return this KeyError message...

Traceback (most recent call last):
File "Q:\Aqua_ArcGIS\WebServiceInfoExtract\test\test7.py", line 23, in <module>
fcnam = b['databases'][0]['datasets'] #scan the JSON and fine the list of feature classes
KeyError: 'databases'

Have any idea?

Re: Filter Services by those which contain .MapServer to get Service Manifest

RobertAkroyd1 — Tue, 15 Nov 2022 08:29:12 GMT

Looks like the JSON for that service did not include a "databases" element, so you should check for that in the code first, e.g.

b = json.loads(a)

if b['databases']:

fcnam = b['databases'][0]['datasets'] #scan the JSON and fine the list of feature classes

Re: Filter Services by those which contain .MapServer to get Service Manifest

JaymieCroucher — Tue, 15 Nov 2022 09:51:01 GMT

Sorry I only recently managed to get this working in anger. The below provides a CSV output of the following:

Each Feature Class and sourced database provided within a service
The Service URL and Name
The Portal URL and ID

import json
import csv
import os
from arcgis.gis import *

#Step 1 - Import all libraries and setup connection to portal and location of csv

gis = GIS("Portal URL", username, password,verify_cert=False,)
outputFile=r"\\Output CSV Location"
print("Successfully logged in as: " + gis.properties.user.username)

#Step 2 - wipe the CSV and set header values

with open(outputFile, 'w', newline='') as ResultFile:
wr = csv.writer(ResultFile)
header = 'Portal Item', 'Portal Item ID','Service URL','Portal URL','DB Name', 'Feature Class Name'
wr.writerow(header)

#Step 3 - Find hosting server and list and join all services across folders

servers = gis.admin.servers
s = servers.list()
server1 = s[0] #isolate portal server in list
services_1 = server1.services.list(folder="Server Folder Name 1") #list the services within the folder "root"
services_2 = server1.services.list(folder="Server Folder Name 2")
services_3 = server1.services.list(folder="Server Folder Name 3")
services_4 = server1.services.list(folder="Server Folder Name 4")
services = services_1 + services_2 + services_3 +services_4

#Step 4a - Isolate only MapServer services and print the manifest for each service
#Step 4b - Extract service properties from the Portal
#Step 4c - Sift through JSON from manifest and extract feature class and database
#Step 5 - Write out many to one relationship of feature class to service to portal item

for service in services:
if service.type == "MapServer":
ii = service.iteminformation
manifest = ii.manifest
a = json.dumps(manifest)
b = json.loads(a)
piid = service.properties.portalProperties.portalItems
pin = service.properties.serviceName
piidi = piid[0]['itemID']
piidiurl = "Portal Map Viewer URL" + piidi
fcnam = b['databases'][0]['datasets'] #scan the JSON and fine the list of feature classes:
dbnam = b['databases'][0]['onServerName'] # scan the JSON and find the DB name
for x in fcnam:
with open(outputFile, 'a',newline='') as ResultFile:
wr = csv.writer(ResultFile)
ContentRow = pin,piidi,service,piidiurl,dbnam,x['onServerName']
wr.writerow(ContentRow)
loops = loops+1

print("Successfully wrote " +str(loops) + " records to the manifest CSV")

Re: Filter Services by those which contain .MapServer to get Service Manifest

HaThach1 — Tue, 15 Nov 2022 15:13:36 GMT

I got the same error after updating the script.

Re: Filter Services by those which contain .MapServer to get Service Manifest

HaThach1 — Tue, 15 Nov 2022 15:30:49 GMT

Hi Jamie,

I tried your script as well and got

AttributeError: 'PropertyMap' instance has no attribute 'portalProperties'

even if I comment out variable piid, now I'm getting the same error as before relate to

KeyError: 'databases'

Re: Filter Services by those which contain .MapServer to get Service Manifest

JaymieCroucher — Tue, 15 Nov 2022 15:59:02 GMT

Do you get a list of services from the hosting server?

Have you manually checked the JSON manifest through Server Manager for a specific service?

(Should look like: https://resources.arcgis.com/en/help/server-admin-api/serviceManifest.html. Gained through Server Manager) Example URL here: https://portal.domain/server/admin/services/ServerFolder/ServiceName.MapServer/iteminfo/manifest/manifest.json

Re: Filter Services by those which contain .MapServer to get Service Manifest

HaThach1 — Tue, 15 Nov 2022 18:06:22 GMT

I didn't get pass this error

AttributeError: 'PropertyMap' instance has no attribute 'portalProperties'

As for the JSON manifest check, I was able to constructed the URL from your example and see the JSON manifest of a specific service.