topic Re: AGS admin scripting help in Python Questions

AGS admin scripting help

JamesCrandall — Thu, 28 Jul 2016 13:15:21 GMT

I'm trying to build some simple service checks (map, feature and gp services) that are published on our ArcGIS Server site(s). Example: Check a folder for stopped services—Documentation (10.3 and 10.3.1) | ArcGIS for Server is what I'm attempting to implement but having difficulty with the request at fetching the token.

I've been provided with the serverName and serverPort parameters (lines 22 and 23 of the sample code) by our AGS admin but I keep tripping the error at line 137. If I print the response.status it shows a response of 302 and response.reason is "Found".

Any input or success implementing this script?

Re: AGS admin scripting help

JamesCrandall — Thu, 28 Jul 2016 14:53:51 GMT

I ditched using the REST API calls (it's infuriating to navigate and determine all of the correct required server parameters). Simply opted for simplicity and accomplished this with urllib.open and properly setting up each individual url, taking care in constructing each one correctly.

The REST API is overly complicated for my needs.

Re: AGS admin scripting help

JamesCrandall — Thu, 28 Jul 2016 16:31:35 GMT

I may revisit the REST API as there are some other complications to overcome with the approach I took.

The issue with the REST API example is that it doesn't implement the requests library and instead uses urllib. Normally not an issue, however when dealing with https and TLS & SSL things start to just not work as originally written! This sends you down an entirely different path --- I may just try to re-write the sample using requests instead.

Re: AGS admin scripting help

RebeccaStrauch__GISP — Thu, 28 Jul 2016 16:50:36 GMT

Just to make sure you aren't reinventing the wheel, have you checked out

Downloadable tools for ArcGIS Server administration | ArcGIS Blog

http://www.arcgis.com/home/item.html?id=12dde73e0e784e47818162b4d41ee340

Re: AGS admin scripting help

JamesCrandall — Thu, 28 Jul 2016 18:03:11 GMT

Thanks!

Have you had a chance to setup the toolkit against HTTPS? I'd think we'd run into the same issues I'm hitting with the example REST API script.

Re: AGS admin scripting help

JamesCrandall — Thu, 28 Jul 2016 18:39:41 GMT

This looks like a really comprehensive tool set! But it's a bit much for what we need I think. To go from just running a simple script to check what services are stopped/running to this is pretty extreme.

I'm not sure why the example script doesn't play nice with https. Seems like something do-able to overcome.

Re: AGS admin scripting help

RebeccaStrauch__GISP — Sat, 11 Dec 2021 22:21:58 GMT

mine runs using the machine name as input and runs again sucure and non-secure services. I run the tool in the toolbox, then copy the snippet for input into my own script....the only thing I don't like just using this as default is the admin password is part of the snippet (in the results tab too), so be aware of that.

sample of the script...with fake folder names for demonstration....

arcpy.StopStartService(Server_Name="<machinename>",Port="6080",
  Admin_User="<adminUser>",Admin_Password="<adminPsw>",
  Stop__Start__Delete="Stop",
  Service_List="some_gp//ExtractData.GPServer;print//dwcGeoPDF.GPServer;our_public//GMUSubunits.MapServer;our_secure//uculines.MapServer")

If you run it once "manually" with Stop, then you can copy and change to Start to you don't always start up the services you don't want started.

Re: AGS admin scripting help

KevinHibma — Thu, 04 Aug 2016 13:42:05 GMT

I just came across this post and realize you might have a path forward.

Just a note though - I've been trying to make time to refresh the toolkit. That includes introducing proper HTTPS support. If you're interested in it before its "done", its in a branch here: GitHub - khibma/ArcGISAdministrationToolkit at devupdates

I'm close to being done. I just need to do some more testing. After that I'll update the arcgis.com sample with the newest and greatest.

Re: AGS admin scripting help

JamesCrandall — Thu, 04 Aug 2016 13:50:23 GMT

That's great, Kevin! Thanks!

Re: AGS admin scripting help

RebeccaStrauch__GISP — Thu, 04 Aug 2016 15:39:42 GMT

Hi Kevin,

This may not be the place to ask this, but since you mentioned you are working on it., is there a way that the commands can use an .ags administrator (or publisher) connection file, rather than be prompted and/or hardcoding the credentials? It bothers me that the info ends up in the results. Maybe that is what you are attempting (or attempting to hide) in the SSL updates??

I haven't done much directly with the ArcGIS Server REST API myself. Just started looking at it, and the admin tools (the current release, not the gishub version, which I will try out now). I'm trying to work on the workflow for when I need to update the source FGDB being used....i.e

check for any services that access the FGDB
check for those that are currently STARTED
shut those down
rename/replace FGDB
start ONLY those stopped in step 2

Problems with the current tool is:

it lists ALL the services, whether started or stopped already, so if I stop all the ones I need, and duplicate the list, it may start some that I had stopped on purposes already
no easy way at this time to tell those I need to shut down (i.e. filter for "services using source"

Let me know if you need me to put this info in GitHub as a comment. I haven't posted much to github (more of a downloader still)

Re: AGS admin scripting help

KevinHibma — Thu, 04 Aug 2016 16:30:29 GMT

Hey Becky - The ability to use the .AGS file would be an excellent enhancement. I actually really like that idea as it hides the password from being inside your scripts. The only problem: from a pure Python code perspective, I have no way to get at the contents of the AGS file, nor is there any arcpy method that could consume the file to give me what I need. Basically we'd have to do something in the core software to assist in using the connection file to help the tools get what they need. I'll have a conversation about that with some team members.

I understand the not knowing if something is stopped or started from the tool UI. Would it simply be enough to show in the UI something like "MyService [STOPPED]" or "Service2 [STARTED]" ? Or are you after an actual filter to only show certain services? I havent wrote any logic to filter, but if I dont consider it now, I probably wont look at making these tools "better" for another 4 years after this go.

Re: AGS admin scripting help

RebeccaStrauch__GISP — Sat, 11 Dec 2021 22:22:01 GMT

Kevin,

re: the .AGS file. I thought there were other "tools" that could use this as input already. Not necessarily pure Python code, by maybe arcpy code?? I'll have to look to see if I can find any samples, or things that maybe I have used.

Kevin Hibma re: the filter. When you startup ArcGIS Server, it does remember what was started when you shut AGS, so that information is stored somewhere. But obviously I'm not necessarily wanting to shut the AGS site down just to replace the source for these dozen or so services. BTW - because my process recreates a FGDB and derived datasets each time (from a master SDE database), I can't use replicate and need to replace it each time.

I'll have to work on the filter part myself I think....

maybe use a inventory of all data sources withing all the .mxds being used by the services, search for the \\machinename\fgdbpath and/or d:\fgdbpath in that inventory. Write a list of all these folders\services
then use something like the sample on Manage Service utility—Documentation (10.4) | ArcGIS for Server page to find all the the STARTED services

The following example lists all the services in the site.
<Python installation location>\python.exe "C:\Program Files\ArcGIS\Server\tools\admin\manageservice.py" -u admin -p admin -s http://gisserver.domain.com:6080 -l

and it looks like the the key here is the

-l Lists the services on the GIS server and displays whether they are stopped or started.

or use Example: Check a folder for stopped services—Documentation (10.4) | ArcGIS for Server but change it to STARTED instead of STOPPED (??). anyway, if I can just check those in the list I created, and updating the list for those that are currently STARTED

run the stop against my filtered list
rename/replace....
run the start on same filtered list

I'm still looking around to see if anyone has already put a workflow like this together, but I think I'm going to have to write it myself. But again, the real snag here are the admin credentials. This is all part of a python addin toolbar, and not something I want to hardcode into my addin. This addin does take advantage of an .SDE connection file already, so being able to do the same with the .AGS connection file would be ideal.

So, I'm open to any suggestions re: getting the credentials input without exposing the info in the script, addin, or results tab (or anything that can grab the clear-text info). Thanks much for the tools so far....I have used them often (just a pain to have to manually figure out my filtering..and of course the user/pass issue.

btw - my sincere apologies James Crandall for hijacking the thread, but hopefully this is something you are interested in too.

Re: AGS admin scripting help

JamesCrandall — Sat, 11 Dec 2021 22:22:04 GMT

No problem, hijack away!

Here's an idea to keeping credentials out of things. Well, sort of.

You could just setup a simple .ini file to hold any credentials and then use base64 library in the python scritping. It's not a 100% secure approach, but it'd keep any hardcoded credentials out of the codebase and some low-level secuity using base64 to encode/decode that will essentially hide any variable values from view that might be set. Like so:

*** create a "settings.ini" file and save it in the same directory as the .py script
[AGS]
USER = someusername
PASS = somepassword

The implement base64 and ConfigPaser in your .py scripting:

import ConfigParser
import base64
# Find and gather settings from the settings.ini file
localPath = sys.path[0]
settingsFile = os.path.join(localPath, "settings.ini")
config = ConfigParser.ConfigParser()
config.read(settingsFile)

# Get encoded credentials
uname_encoded = base64.b64encode(config.get( 'AGS', 'USER'))
pwd_encoded = base64.b64encode(config.get('AGS', 'PASS'))

# use the encoded credentials for any login parameters
# for example, build the URL with credentials for the REST API
agsurl = 'https://yourAGSaddress'
site = '/arcgis'
URL = agsurl + site + "/rest/services"
      
# obtain a token  
referer = agsurl
query_dict = { 'username': base64.b64decode(uname_encoded), 'password': base64.b64decode(pwd_encoded), 'referer': referer }  
query_string = urllib.urlencode(query_dict)
url = agsurl + site + "/tokens/generateToken"
token = json.loads(urllib.urlopen(url + "?f=json", query_string).read()) 

#...blah blah blah you get the idea

Re: AGS admin scripting help

KevinHibma — Fri, 05 Aug 2016 13:07:04 GMT

This is a great suggestion. Having user/password saved in plain text is just bad form. And sadly, I've encouraged that in places (GitHub - arcpy/update-hosted-feature-service: Update and overwrite a hosted feature service on ArcGIS.com ) Maybe I'll try to update that sample with your approach. And, yes, its still very possible for someone to find out the credentials. However, any sort of security is better than no security. Thanks, James.

Re: AGS admin scripting help

KevinHibma — Fri, 05 Aug 2016 13:33:29 GMT

There are some tools/functions that make use of .AGS files. There is a big difference between "those tools" and the tools in the Administration toolkit. The built-in, system tools have access to all the internal workings of ArcGIS. The tools I've written in Python dont have the same level of 'arcgis' access. Simply put, the tools I wrote dont rely on ArcGIS/arcpy, and even if they did, they couldn't make use of a .AGS file to accomplish the end goal of 'administrative tasks' on the server. Those 'hooks' are not exposed in the software to 'python developers'.

You're right about the stopped|started information being held somewhere. However the interaction with the tools and the server itself is very short lived. The tools only are aware and talk to the server during their execution. As such, any tool or script needs to get all the information from the server and perform the action during that session. So the tools I wrote would need to query the server, get that information, and then perform an appropriate action. The LIST script you reference is essentially the same code I have in the STOP | START | DELETE tool. This action of getting the services and their state needs to happen each time the tool is run.

The idea James offers below is a great method of hiding credentials from plain view. It isn't 100% secure (really any solution that requires you to save credentials on the client machine isn't secure). But its better plain text!

Re: AGS admin scripting help

RebeccaStrauch__GISP — Wed, 10 Aug 2016 02:10:00 GMT

Hi Kevin Hibma ,

I'm making (slow) progress using the sample James Crandall provided above (Many thanks for that James!)

Quick (?) question(s)...

I was looking at the samples for ListingMapServices ArcGIS Help (10.2, 10.2.1, and 10.2.2) but only seems to pass the "unsecure" services, even if I pass the token. Your "Stop, Start, Delete services" will list them all, but is not filtered for those that are STARTED.

How can I get a script to return all of the services (secured or not), but filtered for only those that are STARTED? Kind of a combo of the above....but also almost opposite of both.
I might have better luck once I have it as a tool, but I'm still testing as snippets in the python window

Again, long term goal (simplified)...stop only those services that are accessing the FGDB that I need to update..replace the FGDB...then start just those back up. All while keeping credentials secure. Simple...right. lol. I may need to look at the link James had at the beginning of the thread to see if I can do the opposite of what it is supposed to do.

On that same "secure" thought....is there anyway to block things being written to the results table....or to programatically remove entries from the script? did decide that rather than using the "agsurl " as shown in James' script, if I use the local server name and port 6080, that should prevent access from outside the network (and thinking maybe a local server account??).

Re: AGS admin scripting help

Anonymous User — Fri, 12 Aug 2016 09:24:52 GMT

i am using current version of AGS admin tool with the help of this tool I am to list down all stop services without error.

and also create one Python script to generate reports of all map services.
if you require then i will share with you.

thanks.

Re: AGS admin scripting help

JamesCrandall — Sat, 11 Dec 2021 22:22:07 GMT

We just have a simple need to check services to see STOPPED/STARTED status --- that's really it. Couple of reasons why:

1. To build a dashboard that can be displayed on a monitor hanging on the wall. Pretty simple .aspx page or something like that showing Red & Green for stopped/started services.

2. We have applications developed with the JavaScript API that other departments within the organization build and these utilize our ArcGIS Server services (map, feature & GP services). With the simple code below we published as a GP service that those applications can check against to see status and handle the appropriate methods if at runtime of their application it experiences a stopped service. Again, pretty simple yet has a lot of value.

Part of the GP service that checks service status:

def reviewRunning(token):

    query_dict = { "f": "json", "token": token['token'] }  

    # query your services url  
    jsonResponse = urllib.urlopen(URL, urllib.urlencode(query_dict))
    rooturl = agsurl + site + "/admin/services"
    rootresp = json.loads(urllib.urlopen(rooturl + "?f=json", urllib.urlencode(query_dict)).read())
    fldrs = rootresp['folders']
    for fldr in fldrs:
       fldrurl = agsurl + site + "/admin/services/" + fldr  

       query_dict = { "f": "json", "token": token['token'] }   
       svcresp = json.loads(urllib.urlopen(fldrurl + "?f=json", urllib.urlencode(query_dict)).read())
       svcs = svcresp['services']
       for svc in svcs:
           svcname = svc['serviceName']
           svctype = svc['type']

           query_dict = { "f": "json", "token": token['token'] }

           svcstatusurl =  fldrurl + "/" + svcname + "." + svctype + "/status"
           svcstatusresp = json.loads(urllib.urlopen(svcstatusurl + "?f=json", urllib.urlencode(query_dict)).read())

           svcstatus = svcstatusresp['realTimeState']
           print svcname, svctype, svcstatus

Re: AGS admin scripting help

SubuSwaminathan1 — Fri, 12 Aug 2016 16:07:14 GMT

I agree with Rebecca that being able to hide/encrypt or encapsulate the admin user credentials in a secure fashion is the most important piece that is missing. Once that is in place, being able to able to access secured services, https, service status and all others all become more useful. In fact looking at the this thread and the contributions, most of the rest has been achieved and folks are willing to share with the community.

Thanks

Subu

Re: AGS admin scripting help

SubuSwaminathan1 — Fri, 12 Aug 2016 16:22:48 GMT

James,

Have you found that services need time to stop or that ArcGIS Server needs to recover before the next stop or start request is sent in. I was testing the stop functionality with a large number of services and soon found that I need to introduce a delay if I can expect ArcGIS Server to respond to each request in a reliable fashion. So this meant that I had to increase the token's expiration time as well as insert a time delay to ensure that the server is ready to respond. Otherwise I get an urllib error saying that the process has timed out. Still early days in terms of using this in production in our org.

Thanks

Subu