topic Re: Start/Stop Secure Services in 10.6.1 in ArcGIS Enterprise Questions

Start/Stop Secure Services in 10.6.1

AmyRoust — Thu, 21 Mar 2019 19:42:26 GMT

Prior to 10.6.1, I was using the ArcGIS Server Admin Toolkit to stop and start services in bulk. That toolkit has depreciated and does not appear to work in 10.6.1. (It could be that I'm making an error with the parameters, but I don't think I am) Then I found this:

Example: Stop or start all services in a folder—ArcGIS Server Administration (Windows) | ArcGIS Enterprise

The above script assumes that you are using port 6080, but ours are on 6443. I tried to change the parameter in the same script from 6080 to 6443, but that did not work. Again, I'm not ruling out the possibility that I'm entering the wrong variables, but the error message says that the server is forcibly closing the connection to port 6443 when I run it.

In scouring the forums, I've seen comments that suggest that A) you cannot start/stop services using Python if said services are secured (https), and B) the Esri is intentionally not maintaining or developing tools to allow you to batch start/stop services anymore because they want you to do it through Server Manager/Portal/ArcCatalog. I know you can't start/stop services in Pro yet, but that it might be coming in a future version.

Can anyone help me fill in the gaps? My ultimate need is to be able to stop all web services at once and then restart them in bulk. I'm using Enterprise 10.6.1, including Portal. I published all of my services to Server using ArcMap 10.6.1.

Re: Start/Stop Secure Services in 10.6.1

MichaelVolz — Thu, 21 Mar 2019 19:54:10 GMT

Amy:

Were you able to stop secured services in an earlier version of AGS? I have never gotten the port 6443 and https services to stop without a python error. Did you ever try to modify AGS security to generate tokens as per below link:

Enable token acquisition through an HTTP GET request—ArcGIS Server Administration (Windows) | ArcGIS Enterprise

I was testing setting up my AGS environment only to accept https protocol, but then I was no longer able to use the script you allude to to stop/start the AGS https services.

Re: Start/Stop Secure Services in 10.6.1

AmyRoust — Thu, 21 Mar 2019 20:05:12 GMT

This is the first time that I've had secured services, so I don't have any prior successes to look back on. (*sad trombone*)

After reviewing my workflows that used bulk stops and starts, I guess I don't have to shut down my web services. I can just tell the database to stop accepting connections and kick out anyone who is currently connected. (This happens in the middle of the night via automated task, so that the end user impact is minimal.) The only use I'll really miss is the automated task that restarted all of my web services at 6 am, just to give us a fresh start if something crashed overnight and didn't restart properly.

Re: Start/Stop Secure Services in 10.6.1

MichaelVolz — Thu, 21 Mar 2019 21:28:20 GMT

Do you have any geocode services? In order to update the associated address locator you would still need to stop/start the service.

Re: Start/Stop Secure Services in 10.6.1

AmyRoust — Thu, 21 Mar 2019 21:47:02 GMT

Good point. I do have one locator so that people can search by intersection. Esri probably wants us to move entirely to string searches within layers, but unless I generate an intersection point layer, I still need that geocoder.

Re: Start/Stop Secure Services in 10.6.1

MichaelVolz — Mon, 25 Mar 2019 15:41:40 GMT

I believe the security changes in ArcGIS Server 10.7 (only TLS 1.2 compatibility) will require that all AGS services will be administered with https protocol. You might get away with http and port 6080 at 10.6.1, but you will need https syntax to work in 10.7.

I hope someone (most preferably from ESRI) can provide guidance on the configuration needed to get https protocol to work with the link you provided above:

Example: Stop or start all services in a folder—ArcGIS Server Administration (Windows) | ArcGIS Enterprise

Re: Start/Stop Secure Services in 10.6.1

JonathanQuinn — Sun, 12 Dec 2021 03:25:28 GMT

Programmatically interacting with any administrative API is and will continue to be an integral part of the software. HTTPS doesn't necessarily mean they're secured, at least in the same sense that you'd say you need a token to reach them. It simply means that the traffic to reach them is encrypted.

In order to reach an HTTPS endpoint via Python, the certificate used for the endpoint needs to be trusted by Python or an untrusted certificate needs to be ignored. This isn't specific to Esri, ArcGIS Server, arcpy, etc. Server, even at 10.7, can be HTTPS and HTTP, but by default on a new site, it is HTTPS. Ideally, the certificate is already trusted by Python, or with a bit of work, you can configure Python to trust your own certificates:

https://stackoverflow.com/questions/39356413/how-to-add-a-custom-ca-root-certificate-to-the-ca-store-used-by-pip-in-windows

The following script uses urllib and urllib2 instead of the httplib library to make web requests and disables certificate validation:

import urllib, urllib2, json,ssl

baseURL = 'https://server.domain.com:6443/arcgis'
username = 'admin'
password = 'admin'
whatToDo = "start"

def openURL(url,params=None, protocol=None):
    try:
        if params:
            params.update(dict(f="json"))
        else:
            params = dict(f="json")
        if protocol:
            encoded_params = str.encode(urllib.urlencode(params))
            encoded_params = encoded_params.decode("utf-8")
            url = "{0}?{1}".format(url, encoded_params)
            request = urllib2.Request(url)
            #Ignore certificate validation
            sslContext = ssl._create_unverified_context()
            request.add_header('referer',baseURL)
            response = urllib2.urlopen(request, context=sslContext)
        else:
            encoded_params = str.encode(urllib.urlencode(params))
            request = urllib2.Request(url)
            #Ignore certificate validation
            sslContext = ssl._create_unverified_context()
            request.add_header('referer',baseURL)
            response = urllib2.urlopen(request,encoded_params,context=sslContext)
        decodedResponse = response.read().decode('utf-8')
        jsonResponse = json.loads(decodedResponse)
        return jsonResponse
    except urllib2.HTTPError as e:
        return e
    except urllib2.URLError as e:
        return e
    except Exception as e:
        print(e)
        
def createToken(baseURL,username,password):
    tokenURL = "{}/tokens/generateToken".format(baseURL)
    params = {"username":username,
              "password":password,
              "client":'referer',
              "referer":baseURL}
    token = openURL(tokenURL,params)['token']
    return token

'''Function to stop or start services'''
def startStopService(serviceURL):
    serviceName = urllib2.urlparse.urlparse(serviceURL).path.split("/")[-1]
    resp = openURL("{}/{}".format(serviceURL,whatToDo),params)
    if "status" in resp and resp['status'] == 'success':
        print("Successfully {} {}".format(msg,serviceName))
    else:
        print("Unable to {} {}.\n {}".format(whatToDo,serviceName,resp))    

token = createToken(baseURL,username,password)
params = {"token":token}
servicesURL = '{}/admin/services'.format(baseURL)
root = openURL(servicesURL,params)
if whatToDo == "stop":
    msg = "stopped"
else:
    msg = "started"
for service in root['services']:
    serviceURL = '{}/{}.{}'.format(servicesURL,service['serviceName'],service['type'])
    startStopService(serviceURL)
for folder in root['folders']:
    if not folder in ['System','Utilities']:
        folderURL = "{}/{}".format(servicesURL,folder)
        folderServices = openURL(folderURL,params)
        for service in folderServices['services']:
            serviceURL = '{}/{}/{}.{}'.format(servicesURL,folder,service['serviceName'],service['type'])
            startStopService(serviceURL)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

https://community.netapp.com/t5/Software-Development-Kit-SDK-and-API-Discussions/Python-How-to-disable-SSL-certificate-verification/td-p/113697

You can do this with the requests module as well:

https://stackoverflow.com/questions/15445981/how-do-i-disable-the-security-certificate-check-in-python-requests

I'll reiterate that the preferred solution isn't disabling certificate validation, though.

Re: Start/Stop Secure Services in 10.6.1

MichaelVolz — Mon, 25 Mar 2019 17:42:46 GMT

Jonathan:

Thanks for the python code. Can you tell me which lines of code I would need to modify so that the code does not ignore certificate validation?

Does ArcGIS Server need to be configured based on the following link to properly handle tokens for the code you provided:

Enable token acquisition through an HTTP GET request—ArcGIS Server Administration (Windows) | ArcGIS Enterprise

In the example provided in the original post and even in the code that you provided, is there a way to call the service without providing a port? My org has a trusted certificate setup on our load balancer that is separate from the internal load-balancing of ArcGIS Server, but when I tried to input the parameters for the service with the load-balanced URL and no port it was rejected.

Re: Start/Stop Secure Services in 10.6.1

JonathanQuinn — Sun, 12 Dec 2021 03:25:31 GMT

The baseURL variable on line 3 is where you define the host and optionally port and context to the Server you want to connect to. It doesn't matter if you go through 443, 80, 6443 or 6080 as long as you can reach the Admin API via that URL. For example, you could use your web adaptor URL, https://webadaptor.domain.com/server. You need to make sure that the machine you're running the script on has access to the URL you want to use to connect to.

To validate certificates, get rid of lines 20 and 27 and update line 22 to be

response = urllib2.urlopen(request)

and 29 to be:

response = urllib2.urlopen(request,encoded_params)

You're basically just getting rid of the logic to define the SSL context as "unverified".

Re: Start/Stop Secure Services in 10.6.1

GavinMcDade — Fri, 29 Mar 2019 19:24:09 GMT

Jonathan,

I'm working w/ Mike to revise and test our code, and I've gotten the above to work when bypassing the certificate verification. However, when I comment out and alter the code as you've indicated above, I receive the following Python error (ignore line numbers, as they differ from your raw script above):

Traceback (most recent call last):
File "Stop_GeocodeService_Cert.py", line 148, in <module>
token = createToken(baseURL,username,password)
File "Stop_GeocodeService_Cert.py", line 118, in createToken
token = openURL(tokenURL,params)['token']
TypeError: sequence index must be integer, not 'str'

For whatever reason, it appears to no longer like the ['token'] parameter... Any ideas why this might be?

Gavin

Re: Start/Stop Secure Services in 10.6.1

JonathanQuinn — Sun, 12 Dec 2021 03:25:34 GMT

Hm, openURL isn't return JSON or a dictionary, but a string or array of some sort. It's probably not a good idea for the token request to assume the response will have the token in it, so in my original example, line 46 should be something like:

resp = openURL(tokenURL,params)
if "token" in resp:
  return resp['token']
else:
  raise Exception("Can't get token: {}".format(resp))
‍‍‍‍‍‍‍‍‍‍‍‍

If you check whether the response as the token property in it, then you can get the token from the response. Otherwise, the script needs to let you know there was an issue with the token request.

Re: Start/Stop Secure Services in 10.6.1

GavinMcDade — Mon, 01 Apr 2019 14:50:48 GMT

Ok, thanks - I'll give this a try. One question, though: I'm assuming that the return resp['token'] line will simply pass the value just the same as the original return token (even though it's setting/using a different variable name)? In other words, the token = createToken(baseURL,username,password) line simply receives the returned value from the function regardless of what it's called?

Gavin

Re: Start/Stop Secure Services in 10.6.1

JonathanQuinn — Tue, 02 Apr 2019 15:42:51 GMT

Right, they're functionally the same. You're getting a dictionary back, ({"token":XXXXX","expires":"YYYY"}), so you just need to extract the "token" variable, however you want to do that.

Re: Start/Stop Secure Services in 10.6.1

KDCGIS — Tue, 02 Jul 2019 04:33:44 GMT

Looking into this I have the same problem as Amy Roust‌ has. I am looking to bulk stop my services to free up my webmap SDE, or I can just stop the AGS windows service or as Amy mentions kick people out of the SDE, but I'd like the ability to shut down a folder's worth of services or all services in AGS via python as I feel it's the more elegant way to do it (could be wrong!). We have a federated server/portal and are running on https for the whole setup. Trying a http URL for the generateToken request site manually results in a TLS error message, whereas trying with https gets me there.

After discussion with my network/server IT admin he is not keen to disable certificate verification and we both agree the solution should not involve that. I see above that you have mentioned configuring to get python to trust your certificates and am not presently sure if I need to, i.e. if python trusts our certificate(s). It looks pretty involved as SSL always does. Is it possible to post example start/stop code with certificate verification? It looks like Michael Volz‌ was working on that.

My services are all published via ArcGIS Pro 2.3 at present, but we are still running on python 2.7 and have not yet ported to python 3.

Re: Start/Stop Secure Services in 10.6.1

MichaelVolz — Tue, 02 Jul 2019 15:20:57 GMT

You do not need to disable certificate verification as you can comment out the line of code that ignores the certificate and it will run if you have a legit certificate in place.

Just comment out line 27

sslContext = ssl._create_unverified_context()

The code was reformatted at my org for more granular stopping and starting of individual services (mostly geocode services that need to be stopped to rebuild address locators), but I think the original code provided by Jonathan Quinn should allow you to stop all services. I also do not have a federated server/portal so I'm not sure if that would impact whether the above code from Jonathan would work.

Re: Start/Stop Secure Services in 10.6.1

JonathanQuinn — Tue, 02 Jul 2019 16:31:25 GMT

When you're working with certificates and Python, I think there are three options:

1) Use a certificate from a trusted root certificate authority, (Digicert, Verisign, etc), that Python trusts by default

2) Configure Python to trust your certificates

3) Disable certificate verification

If you don't want to disable certificate verification, then you need to make sure you're following either 1 or 2, and then as Michael Volz‌ suggested, comment line 20 and 27 but also be sure to remove context=sslContext from lines 22 and 29.

Re: Start/Stop Secure Services in 10.6.1

KDCGIS — Tue, 02 Jul 2019 23:30:37 GMT

Thanks guys, I pretty much arrived at the conclusion after reading all of this that #1 or #2 from your list Jonathan are the ways to go. #3 you have already mentioned is risky. So it seems configuring the certificate side of it is necessary, it adds a chunk of work to the scripting process but should just be a one-time job. I'll look into #1 or #2 with my IT admin.

Re: Start/Stop Secure Services in 10.6.1

MichaelVolz — Mon, 18 May 2020 19:55:54 GMT

Jonathan:

Would this script need some major changes when run with python 3.x that is bundled with Pro as opposed to the current script that is run with python 2.7 that is bundled with ArcMap?

It appears that modules have been renamed and after renaming the modules, it seems like functions from the renamed modules have also been renamed but the documentation I have found on the renamed modules has not specified the function names that can be called.

Re: Start/Stop Secure Services in 10.6.1

JonathanQuinn — Sun, 12 Dec 2021 03:25:36 GMT

Yes, the urllib libraries have changed between versions. You can write your script to support both:

if sys.version_info.major >=3:
    import urllib.request, urllib.error, urllib.parse
    urlEncode = urllib.parse.urlencode
    urlParse = urllib.parse.urlparse
    urlRequest = urllib.request.Request
    urlOpen = urllib.request.urlopen
    httpError = urllib.error.HTTPError
    urlError = urllib.error.URLError
else:
    import urllib, urllib2, shlex, urlparse
    urlEncode = urllib.urlencode
    urlRequest = urllib2.Request
    urlOpen = urllib2.urlopen
    httpError = urllib2.HTTPError
    urlError = urllib2.URLError
    urlParse = urlparse.urlparse‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Then, replace all occurrences of each function with the respective variables. You'd then be able to use the script with Python 3.x or 2.x.

Re: Start/Stop Secure Services in 10.6.1

jorisfrenkel — Tue, 30 Jun 2020 09:08:42 GMT

Nice to know. I just rewrote my own stop/start scripts from Python 2 to Python 3 and then bumped into this thread 😄