Community

Resolved: Testing whether a service is a geoprocessing service, AND if accessing a datasource

1909
3
09-29-2016 04:07 PM
RebeccaStrauch__GISP
by MVP Emeritus
MVP Emeritus
‎09-29-2016 04:07 PM

UPDATE: title changed to better reflect the results needed. 

Bottom line: in python, how can I check whether a data source is being used by a GP service?

I'm working on a script (currently run against AGS10.2.2) that looks at all running services and checks the mxd in, for example:

D:\arcgisserver\directories\arcgissystem\arcgisinput\wc_public\Sublines.MapServer\extracted\v101

to see if a specified file gdb is used in the service. I make a list of STARTED services, make a list of those that are using the datasource, and then stop those services so I can update the fgb and restart the same services.  This is all running fairly well.

The problem is I still get locks if there is a GP service that is accessing the fgdb. (took a while for me to track that down). Since there is no .mxd associated with the GP service in a relative folder, the one suggest (per Kevin Hibma‌ ) would be to dig into the

C:\arcgisserver\config-store\services\wc_gp\ExtractSomeData.GPServer\esriinfo\manifest\manifest.json

file, which means doing a bit of extra work with the json file, (or possibly the .xml file in same location) to see if the path is in there. For example, the .json for the above service is similar to this (the source I'm looking for is green ...but my datapath name is actually a unc path for the D drive, but that I can adjust my search.:)

{"databases":[{"byReference":true,"onServerWorkspaceFactoryProgID":"esriDataSourcesGDB.FileGDBWorkspaceFactory.1","onServerConnectionString":"DATABASE=D:\\data\\wc\\Master_current.gdb","onPremiseConnectionString":"DATABASE=D:\\data\\wc\\Master_current.gdb","onServerName":"Master_current.gdb","onPremisePath":"","datasets":[{"onServerName":"Subunits"},{"onServerName":"UNIT"},{"onServerName":"Regions"}]}],"resources":[{"onPremisePath":"C:\\Users\\usename\\AppData\\Local\\Esri\\Desktop10.2\\Staging\\agsAdmin@myservername_6080 (publisher)\\sddraftToolboxes\\ExtractGMUData.tbx","clientName":"MYSERVERNAME","serverPath":"d:\\arcgisserver\\directories\\arcgissystem\\arcgisinput\\wc_gp\\ExtractGMUData.GPServer\\extracted\\v101\\ExtractGMUData.tbx"}]}

So, my questions are:

  1. has anyone written a python script that searches a GP service manifest.json (or manifest.xml) file for a <datasource> path, gdb in my case? and if so, anything that can be shared? 
  2. does anyone have another way (python script) to query a GP service for a datasource locks?  ......or....when getting a list of STARTED services, is there a way to determine that it is a GP service and not a map service?  I'm actually using a modified function (from Kevin and others) to create this list (and it works for creating my list) 
    EDIT: Think I figured out how to do this part.
    # Get list of Services based on STARTED/STOPPED/or-all
def getServiceList(siteURL, token, statusFilter): 
     # Function to list AGS services based on status: STARTED, STOPPED or all
     #   Requires token 
     # Note: Will not return any services in the Utilities or System folder    
     if not statusFilter == "STARTED" and not statusFilter == "STOPPED":
          statusFilter = ""
     services = [] 
     filteredServices = []
     folder = ''    
     URL = "{0}/admin/services{1}?f=pjson&token={2}".format(siteURL, folder, token)    
     serviceList = json.loads(urllib2.urlopen(URL).read())

     # Build up list of services at the root level
     for single in serviceList["services"]:
          services.append(single['serviceName'] + '.' + single['type'])

     # Build up list of folders and remove the System and Utilities folder (we dont want anyone playing with them)
     folderList = serviceList["folders"]
     folderList.remove("Utilities")             
     folderList.remove("System")
     #folderList.remove("anotherFolderToRemove")

     # working on services in other folders, not root     
     if len(folderList) > 0:
          for folder in folderList:                                              
               URL = "{0}/admin/services/{1}?f=pjson&token={2}".format(siteURL, folder, token)    
               fList = json.loads(urllib2.urlopen(URL).read())

               for single in fList["services"]:
                    services.append(folder + "//" + single['serviceName'] + '.' + single['type'])                    
     if len(services) == 0:
          myMsgs ("No services found")
     else:
          #print ("\nServices on {}:".format(server))   # use to test or debug
          for service in services:
               statusURL = "{0}/admin/services/{1}/status?f=pjson&token={2}".format(siteURL, service, token)
               status = json.loads(urllib2.urlopen(statusURL).read())
               if statusFilter == "":
                    #myMsgs ("  {0} >  {1}".format(status["realTimeState"], service))  # use to test or debug
                    filteredServices = services
               elif status["realTimeState"] == statusFilter:
                    #myMsgs ("  {0} >  {1}".format(status["realTimeState"], service))  # use to test or debug
                    #filteredServices.append((status["realTimeState"], service))  #returns status, good for debug
                    filteredServices.append(service)
     return filteredServices
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
    And my guess is I can query something in that, but haven't dug into that yet....but maybe someone already knows?

I'll be able to grab the correct path to the .json or .xml, in my script when needed.  BTW - I have all the clear memeory and other lock checks that I've gathered already in place.  I manually used compmgmt.msc and fsmgmt.msc to find some, but finally tracked down the final locks using Process Explorer , then stopped one GP service at a time to find the problem GPs.  But would like to get this automated in my script.

Bottom line: in python, how can I check whether a data source is being used by a GP service?

Thanks

Python‌ python snippets‌

EDIT: I think I just figured out how to determine if it is a GP service, so one issue down.  I can do one of two things....one, if I don't find the mxd, most likely a GP service.  But even better, my returned services will end in .GPServer (vs..MapServer).  So that will reduce the number I need to look at the json.  Crossed out message is Remanent, not yet determined.

0 Kudos
3 Replies
JonathanQuinn
by Esri Notable Contributor
Esri Notable Contributor
‎09-30-2016 09:53 AM

Hey Rebecca, can you provide some background on what updates you're making to the feature classes within the FGDB that require you to stop the service?  If you're updating the schema, have you looked into turning off schema locking for the map services?  I'm trying to recall if GP services also have this setting.  If it's data manipulations, then the service doesn't need to be stopped.  You can update individual features or clear out the feature class and append all new records if it's a larger change.  

0 Kudos
RebeccaStrauch__GISP
by MVP Emeritus
MVP Emeritus
‎10-03-2016 11:35 AM

Hi Jonathan,

I'm actually replace the complete FGDB.  Due to processes I run to create the Master_update FGDB, I can not use any replication to the Master_current FGDB.  So in order to replace it, I need to make sure no locks exists, i.e. all services that are accessing the Master_current.gdb must be STOPPED.  I had this working on my dev machine, then tried the production and ran into the GP services.

This is becoming more of a python and json issue than an AGS issue (since I have the rest figured out).  What is coming down to is how can I search the manifest.json to see if a string is present.  I've been looking at several Stackoverflow python/json/string-search threads, but I don't think I'm drilled down far enough yet.

I'm attached a test version of the .json file.  I know I'm close, but struggling with getting all the syntax.  I know the following isn't quite right.   Although it is a "json" file, I'm really just trying to search for a string, not caring what item/section it is in.  I may be trying to make this too difficult.  Simpler suggestions anyone??

import os
import json
import arcpy

from pprint import pprint 
gpJsonPath = r"c:\temp\manifest.json"

searchTerm = "Master_current.gdb"   # this was wrong in my original post, had _update instead of _current
jsonFile = open(gpJsonPath, 'r')
jsonData = json.load(jsonFile)
jsonFile.close()
pprint(jsonData)


jsonFile.close()
jsonDatabases = jsonData['databases']


pprint(jsonDatabases)
pprint(jsonDatabases[0])

#   searchTerm in jsonDatabases    #  this obviously doesn't work.‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

 I've tried a few loops and will continue to try, but looking for better solutions as I continue to look.  Thanks

--btw - I'll probably change the title of this thread to make it more focused on my question, but didn't want to change mid-stream.  thanks.

manifest.json.zip
0 Kudos
RebeccaStrauch__GISP
by MVP Emeritus
MVP Emeritus
‎10-04-2016 03:57 PM

For anyone interested, I was able to solve my issue. Although not quite done with the full script (need to merge one of these in to  main code), I am now able to search the manifest.json file for a given source.  My initial issue was user error (i.e. "me") in that I was searching for the incorrect string.  Duh!.   So, once that was figured out, I realized there are many ways to accomplish this. With the definite possibility that this is TMI (too much info), I'm including a script that I used to most combinations of the following things to check:

  1. good vs incorrect/bad .gdb to test, 
  2. using == vs IN,
  3. using onServerName vs onServerConnectionString from the json file
  4. and two methods to go thru the file, one with open() close(), the other with the open() in a "with"

So, for what it's worth:

import os, sys
import json
import arcpy
from pprint import pprint 
gpJsonPath = r"c:\temp\manifest.json"
searchTerm = "Master_current.gdb"
searchTerm2 = "Master_update.gdb"

# ########
# option 1
# ########
print("option 1")
jsonFile = open(gpJsonPath, 'r')
jsonData = json.load(jsonFile)
jsonFile.close()
# pprint(jsonData)

# ########  start test 1
print("\n{0}\n Checking the GOOD searchTerm gdb onServerName value\n{0}".format("#" * 60))
checkValue = jsonData['databases'][0]['onServerName']
print("\n  check for GOOD searchterm EQUAL to onServerName")
if searchTerm == checkValue:
     print("      ==.. {0} is being used, add to list of services to stop".format(searchTerm))
else:
     print("result 1a: {0} not being used...continue to next service")
print("\n  check for GOOD searchterm IN onServerName")
if searchTerm in checkValue:
     print("      IN.. {0} is being used, add to list of services to stop".format(searchTerm))
else:
     print("result 1b: {0} not being used...continue to next service")
# ######## end test1

# ######## start test 2
print("\n{0}\n Checking the GOOD searchTerm gdb onServerConnectionString value\n{0}".format("#" * 60))     

checkValue2 = jsonData['databases'][0]['onServerConnectionString']
print("\n  check for GOOD searchterm EQUAL to onServerConnectionString")
if searchTerm == checkValue2:
     print("      ==.. {0} is being used, add to list of services to stop".format(searchTerm))
else:
     print("result 2a: {0} not being used...continue to next service")
print("\n  check for GOOD searchterm IN onServerConnectionString")
if searchTerm in checkValue2:
     print("      IN.. {0} is being used, add to list of services to stop".format(searchTerm))
else:
     print("result 2b: {0} not being used...continue to next service")
# ######## end test 2



# ########  start test 3
print("\n{0}\n Checking the BAD searchTerm gdb onServerName value\n{0}".format("#" * 60))
checkValue = jsonData['databases'][0]['onServerName']
print("\n  check for BAD searchterm EQUAL to onServerName")
if searchTerm2 == checkValue:
     print("      ==.. {0} is being used, add to list of services to stop".format(searchTerm2))
else:
     print("result 3a: {0} not being used...continue to next service")
print("\n  check for BAD searchterm IN onServerName")
if searchTerm2 in checkValue:
     print("      IN.. {0} is being used, add to list of services to stop".format(searchTerm2))
else:
     print("result 3b: {0} not being used...continue to next service")
# ######## end test1

# ######## start test 4
print("\n{0}\n Checking the BAD searchTerm gdb onServerConnectionString value\n{0}".format("#" * 60))     
checkValue2 = jsonData['databases'][0]['onServerConnectionString']
print("\n  check for BAD searchterm EQUAL to onServerConnectionString")
if searchTerm2 == checkValue2:
     print("      ==.. {0} is being used, add to list of services to stop".format(searchTerm2))
else:
     print("result 4a: {0} not being used...continue to next service")
print("\n  check for BAD searchterm IN onServerConnectionString")
if searchTerm2 in checkValue2:
     print("      IN.. {0} is being used, add to list of services to stop".format(searchTerm2))
else:
     print("result 4b: {0} not being used...continue to next service")
# ######## end test 2



# ########################################################################
# option 5 ...suggest from esri. to use "With" when opening file
#  this should auto close open file, so may be better
# ########################################################################

print("\n{0}\n Use With to check GOOD searchTerm gdb onServerName value\n{0}".format("#" * 60))
with open(gpJsonPath) as json_data:
     d = json.load(json_data)
     #print(d['databases'][0]['onServerName'])
     onServerNameValue = (d['databases'][0]['onServerName'])
     onServerConnectionStringValue = (d['databases'][0]['onServerConnectionString'])
     
     # check 5a          
     print("\n  check for GOOD searchterm EQUAL to onServerName")
     sourceUsed =  (searchTerm == onServerNameValue)
     if sourceUsed:
          print("      ==.. {0} is being used, add to list of services to stop".format(searchTerm2))
     else:
          print("result 5a: {0} not being used...continue to next service")     
     
     # check 5b
     print("\n  check for GOOD searchterm EQUAL to onServerConnectionStringValue")     
     sourceUsed2 = (searchTerm == onServerConnectionStringValue)
     if sourceUsed2:
          print("      ==.. {0} is being used, add to list of services to stop".format(searchTerm2))
     else:
          print("result 5b: {0} not being used...continue to next service")     

     # check 5c     
     print("\n  check for GOOD searchterm IN to onServerName")
     sourceUsed3 =  (searchTerm in onServerNameValue)
     if sourceUsed3:
          print("      IN.. {0} is being used, add to list of services to stop".format(searchTerm2))
     else:
          print("result 6a: {0} not being used...continue to next service")     
     
     # check 5d
     print("\n  check for GOOD searchterm IN to onServerConnectionStringValue")
     sourceUsed4 = (searchTerm in onServerConnectionStringValue)
     if sourceUsed4:
          print("      IN.. {0} is being used, add to list of services to stop".format(searchTerm2))
     else:
          print("result 6b: {0} not being used...continue to next service")     



          
     print("\n{0}\n Use With to check BAD searchTerm gdb onServerName value\n{0}".format("#" * 60))
     with open(gpJsonPath) as json_data:
          d = json.load(json_data)
          #print(d['databases'][0]['onServerName'])
          onServerNameValue = (d['databases'][0]['onServerName'])
          onServerConnectionStringValue = (d['databases'][0]['onServerConnectionString'])
          
          # check 5a          
          print("\n  check for BAD searchterm EQUAL to onServerName")
          sourceUsed5 =  (searchTerm2 == onServerNameValue)
          if sourceUsed5:
               print("     ==.. {0} is being used, add to list of services to stop".format(searchTerm2))
          else:
               print("result 7a: {0} not being used...continue to next service")     
          
          # check 5b
          print("\n  check for BAD searchterm EQUAL to onServerConnectionStringValue")
          
          sourceUsed6 = (searchTerm2 == onServerConnectionStringValue)
          if sourceUsed6:
               print("      ==.. {0} is being used, add to list of services to stop".format(searchTerm2))
          else:
               print("result 7b: {0} not being used...continue to next service")     
     
          # check 5c     
          print("\n  check for BAD searchterm IN to onServerName")
          sourceUsed7 =  (searchTerm2 in onServerNameValue)
          if sourceUsed7:
               print("      IN.. {0} is being used, add to list of services to stop".format(searchTerm2))
          else:
               print("result 8a: {0} not being used...continue to next service")     
          
          # check 5d
          print("\n  check for BAD searchterm IN to onServerConnectionStringValue")
          sourceUsed8 = (searchTerm2 in onServerConnectionStringValue)
          if sourceUsed8:
               print("      IN.. {0} is being used, add to list of services to stop".format(searchTerm2))
          else:
               print("result 8b: {0} not being used...continue to next service")     


print("done")
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

And for my test file, I would get the following results::

Evaluating 171 lines of code...
option 1

############################################################
 Checking the GOOD searchTerm gdb onServerName value
############################################################

  check for GOOD searchterm EQUAL to onServerName
      ==.. Master_current.gdb is being used, add to list of services to stop

  check for GOOD searchterm IN onServerName
      IN.. Master_current.gdb is being used, add to list of services to stop

############################################################
 Checking the GOOD searchTerm gdb onServerConnectionString value
############################################################

  check for GOOD searchterm EQUAL to onServerConnectionString
result 2a: {0} not being used...continue to next service

  check for GOOD searchterm IN onServerConnectionString
      IN.. Master_current.gdb is being used, add to list of services to stop

############################################################
 Checking the BAD searchTerm gdb onServerName value
############################################################

  check for BAD searchterm EQUAL to onServerName
result 3a: {0} not being used...continue to next service

  check for BAD searchterm IN onServerName
result 3b: {0} not being used...continue to next service

############################################################
 Checking the BAD searchTerm gdb onServerConnectionString value
############################################################

  check for BAD searchterm EQUAL to onServerConnectionString
result 4a: {0} not being used...continue to next service

  check for BAD searchterm IN onServerConnectionString
result 4b: {0} not being used...continue to next service

############################################################
 Use With to check GOOD searchTerm gdb onServerName value
############################################################

  check for GOOD searchterm EQUAL to onServerName
      ==.. Master_update.gdb is being used, add to list of services to stop

  check for GOOD searchterm EQUAL to onServerConnectionStringValue
result 5b: {0} not being used...continue to next service

  check for GOOD searchterm IN to onServerName
      IN.. Master_update.gdb is being used, add to list of services to stop

  check for GOOD searchterm IN to onServerConnectionStringValue
      IN.. Master_update.gdb is being used, add to list of services to stop

############################################################
 Use With to check BAD searchTerm gdb onServerName value
############################################################

  check for BAD searchterm EQUAL to onServerName
result 7a: {0} not being used...continue to next service

  check for BAD searchterm EQUAL to onServerConnectionStringValue
result 7b: {0} not being used...continue to next service

  check for BAD searchterm IN to onServerName
result 8a: {0} not being used...continue to next service

  check for BAD searchterm IN to onServerConnectionStringValue
result 8b: {0} not being used...continue to next service
done

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

And just to wrap this up. this is the version I will most likely use:

import os, sys
import json
import arcpy
from pprint import pprint 
gpJsonPath = r"c:\temp\manifest.json"
searchTerm = "Master_current.gdb"

print("\n{0}\n Use With to check BAD searchTerm gdb onServerName value\n{0}".format("#" * 60))
with open(gpJsonPath) as json_data:
     d = json.load(json_data)
     #print(d['databases'][0]['onServerName'])
     #onServerNameValue = (d['databases'][0]['onServerName'])
     onServerConnectionStringValue = (d['databases'][0]['onServerConnectionString'])
     print("\n  check for GOOD searchterm IN to onServerConnectionStringValue")
     sourceUsed4 = (searchTerm in onServerConnectionStringValue)
     if sourceUsed4:
          print("      IN.. {0} is being used, add to list of services to stop".format(searchTerm2))
     else:
          print("result 6b: {0} not being used...continue to next service")     ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Hope this helps someone own the line.  FYI

0 Kudos