topic Re: Python client for ArcGIS for Server in ArcAPI Questions

Python client for ArcGIS for Server

FilipKrál — Wed, 06 Aug 2014 18:29:35 GMT

Hi all,

What do you think about this new module in arcapi? I called it 'arrest.py', for 'ArcGIS REST Python client'. It is a set of classes that allow you to interact with ArcGIS for Server REST API (only reading though). Check out the examples near the bottom (line 966 and below). The module has no dependencies so you can play with just the single python script.

https://github.com/NERC-CEH/arcapi/blob/feature-arrest/arrest.py‌

The module is in a new branch so far and I would love to get some feedback from you. Is it a good idea? Do you miss anything? Can you add anything?

I hope we can polish it and check it into master arcapi in couple of weeks.

Cheers,

Filip.

Re: Python client for ArcGIS for Server

XanderBakker — Wed, 06 Aug 2014 21:39:08 GMT

Hi Filip,

Thanx for sharing. I will look into it.

Kind regards, Xander

Re: Python client for ArcGIS for Server

larryzhang — Mon, 15 Sep 2014 05:54:04 GMT

Thanks, Filip,

nice idea.

Do you think if it would be good to integrate / build a Python module, which also will go through all GIS services and collect service statistics from ArcGIS Server REST 'logs' and export into Excel or SCV file?

Maybe, you have already this module. If so, please share...

++++++++++++++

The example is available at ArcGIS Help (10.2, 10.2.1, and 10.2.2)

Re: Python client for ArcGIS for Server

larryzhang — Mon, 15 Sep 2014 07:40:25 GMT

it is an example with the chart in the report

Re: Python client for ArcGIS for Server

FilipKrál — Sat, 20 Sep 2014 21:20:48 GMT

Oh, thanks for the tip, Larry. The element of authentication makes it slightly more complex but I'll consider it if I can make a nice simple function out of it.

Re: Python client for ArcGIS for Server

FilipKrál — Wed, 24 Sep 2014 20:22:52 GMT

Hi larry zhang‌, you seem to be able to use the code you referred to but I couldn't make it work.

What version of ArcGIS Server did you use? I thought the /arcgis/admin end point has been changed. Anyhow, I suppose general users don't have credentials for it anyway.

I tried to experiment with the following 10.2.x servers.

http://sampleserver5.arcgisonline.com/arcgis/rest/ (I have no credentials to this one but I couldn't even find the end points)

https://datahub.esriuk.com/arcgis/rest/ (I managed to generate token from https://datahub.esriuk.com/arcgis/tokens/generateToken‌ but I cannot see any admin interface.

Here is the function I tried to get token with. Hopefully I'll figure out some way of using it to access secured services with arcapi/arrest.py.

import urllib2
import urllib
import json
from contextlib import closing

def generateToken(url, username, password):
    params = urllib.urlencode({'username': username, 'password': password, 'client': 'requestip', 'f': 'json'})
    headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}
    req = urllib2.Request(url, params, headers)
    with closing(urllib2.urlopen(req)) as res:
        dta = res.read()
    jsn = json.loads(dta)
    return jsn.get('token')
       
username,password = '***', '***'
url = "https://datahub.esriuk.com/arcgis/tokens/generateToken"
token = generateToken(url, username, password)

Regards,

Filip.

Re: Python client for ArcGIS for Server

AlexanderNohe1 — Thu, 25 Sep 2014 00:05:44 GMT

Hi Filip,

ESRI in addition also produces their own Python API for accessing REST Services from ArcGIS Server and ArcGIS Online. The API for ArcGIS Server can be found here:

Esri/ArcREST · GitHub

The API for ArcGIS Portal (ArcGIS Online) can be found here:

Esri/portalpy · GitHub

If you find something that ArcREST cannot do that arrest.py does, it may be beneficial to submit it to the ESRI repository. In addition, you may also want to consider a multitude of possible APIs to access REST services as I have found that one will support things that the others will not.

I hope this helps!

Re: Python client for ArcGIS for Server

larryzhang — Sun, 28 Sep 2014 05:17:58 GMT

Thanks a lot, dear all, Greeting, Filip, pls refer to the blog at http://isbullsh.it/2012/06/Rest-api-in-python/ , which indicates that urllib2 is not right way to go... Hi, Alex, It looks that the links like https://github.com/esri/arcrest not working to me. Not sure why?

Re: Python client for ArcGIS for Server

larryzhang — Tue, 28 Oct 2014 07:11:13 GMT

Filip,

Can you check the example codes to collect statistics of all services, which is available at ArcGIS Help (10.2, 10.2.1, and 10.2.2) ?

Re: Python client for ArcGIS for Server

FilipKrál — Sun, 12 Dec 2021 10:59:49 GMT

Hi Larry,

Thanks for the tip about urllib2. It is true that working with urllib2 is a bit awkward but exactly this awkwardness is removed if you use my arrest.py. More serious issues I've had with urllib2 were related to requesting data from https with self signed certificate. I would prefer the requests package but I didn't want to add a dependency on an external package. Is requests part of core Python in 3.x?

I also looked at the script to collect stats of all services you referred to and translated it into a function below. I could not test it because I don't have admin credentials to any server I can reach at the moment. Can you check if it works?

Filip.

"""Queries the logs and writes statistics on map service activity.
Modified version of script available on ArcGIS Help:
http://resources.arcgis.com/en/help/main/10.2/index.html#/Example_Derive_map_service_statistics_from_the_ArcGIS_Server_logs/0154000005vw000000/
To try this example, follow these steps:
1) Publish several map services (without defining tile caches for them).
2) Set the ArcGIS Server log level to FINE.
3) Make some draw requests of your services by adding them to ArcMap etc.
4) Run this script and examine the resulting text file.
"""

import httplib, urllib, json
import sys, time

def collect_services_statistics(server_name, output_file, username, password, hours=24, port=6080):
    """Query ArcGIS Server logs and write a summary of all map service draws.
    server_name -- e.g. 
    output_file -- path of the text file where the statistics should be written
    username, password -- credentials for the server admin rest interface
    hours=24 -- result will include summary for this number of the last hours
    port=6080 -- arcgis server port number
    """
    
    def assertJsonSuccess(data):
        """Check that the input JSON object is not an error object."""
        obj = json.loads(data)
        if 'status' in obj and obj['status'] == "error":
            print "Error: JSON object returns an error. " + str(obj)
            return False
        else:
            return True
    
    def getToken(username, password, server_name, port):
        """Generate a token given username, password and the adminURL."""
        # Token URL is typically http://server[:port]/arcgis/admin/generateToken
        tokenURL = "/arcgis/admin/generateToken"
        # URL-encode the token parameters
        params = urllib.urlencode({'username': username, 'password': password, 'client': 'requestip', 'f': 'json'})
        headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}
        
        # Connect to URL and post parameters
        httpConn = httplib.HTTPConnection(server_name, port)
        httpConn.request("POST", tokenURL, params, headers)
        
        # Read response
        response = httpConn.getresponse()
        if (response.status != 200):
            httpConn.close()
            raise Exception("Unable to fetch token, check the URL and try again.")
        else:
            data = response.read()
            httpConn.close()
            
            # Check that data returned is not an error object
            if not assertJsonSuccess(data):            
                return
            
            # Extract the toke from it
            token = json.loads(data)       
            return token['token']
    
    
    millisecondsToQuery = hours * 60 * 60 * 100 # 86400000 is 1 day
    hitDict = {}
    
    # Get a token
    token = getToken(username, password, server_name, port)
    if token == "":
        raise Exception("Could not generate a token using the credentials.")
    
    # Construct URL to query the logs
    logQueryURL = "/arcgis/admin/logs/query" # is this going to work in future?
    startTime = int(round(time.time() * 1000))
    endTime = startTime - millisecondsToQuery
    logFilter = "{'services':'*','server':'*','machines':'*'}"
    
    params = urllib.urlencode({
        'level': 'FINE',
        'startTime': startTime, 
        'endTime': endTime, 
        'filter':logFilter, 
        'token': token, 
        'f': 'json'
    })
    
    headers = {
        "Content-type": "application/x-www-form-urlencoded",
        "Accept": "text/plain"
    }
    
    # Connect to URL and post parameters    
    httpConn = httplib.HTTPConnection(server_name, port)
    httpConn.request("POST", logQueryURL, params, headers)
    
    # Read response
    response = httpConn.getresponse()
    if (response.status != 200):
        httpConn.close()
        raise Exception("Error while querying logs.")
    else:
        data = response.read()


        # Check that data returned is not an error object
        if not assertJsonSuccess(data):          
            raise Exception("Error returned by operation. " + data)
        
        # Deserialize response into Python object
        dataObj = json.loads(data)
        httpConn.close()


        # Need these to calculate average draw time for an ExportMapImage call
        mapDraws = 0
        totalDrawTime = 0 
        
        # Iterate over messages        
        for item in dataObj["logMessages"]:            
            if item["message"] == "End ExportMapImage":

                elapsed = float(item["elapsed"])
                keyCheck = item["source"]

                if keyCheck in hitDict:
                    stats = hitDict[keyCheck]
                    stats[0] += 1 # Add 1 to tally of hits
                    stats[1] += elapsed # Add elapsed time to total elapsed time
                else:
                    # Add key with one hit and total elapsed time
                    hitDict[keyCheck] = [1, elapsed]

        # Open text file and write header line       
        with open(output_file, "w") as summaryFile:
            header = "Service,Number of hits,Average seconds per draw\n"
            summaryFile.write(header)

            # Read through dictionary and write totals into file 
            for key in hitDict:
                # Calculate average elapsed time
                totalDraws = hitDict[key][0]
                totalElapsed = hitDict[key][1]
                avgElapsed = 0

                if totalDraws > 0:
                    #Elapsed time divided by hits
                    avgElapsed = (1.0 * (totalElapsed / totalDraws))

                # Construct and write the comma-separated line   
                line = key + "," + str(totalDraws) + "," + str(avgElapsed) + "\n"
                summaryFile.write(line)
        
        return summaryFile
    
if __name__ == "__main__":
    server_name = 'http://sampleserver5.arcgisonline.com'
    output_file = 'c:/temp/summary.csv'
    username, password = "that_is_what", "i_do_not_know"
    collect_services_statistics(server_name, output_file, username, password, hours=24, port=6080)