Hi all,
What do you think about this new module in arcapi? I called it 'arrest.py', for 'ArcGIS REST Python client'. It is a set of classes that allow you to interact with ArcGIS for Server REST API (only reading though). Check out the examples near the bottom (line 966 and below). The module has no dependencies so you can play with just the single python script.
https://github.com/NERC-CEH/arcapi/blob/feature-arrest/arrest.py
The module is in a new branch so far and I would love to get some feedback from you. Is it a good idea? Do you miss anything? Can you add anything?
I hope we can polish it and check it into master arcapi in couple of weeks.
Cheers,
Filip.
Hi Filip,
Thanx for sharing. I will look into it.
Kind regards, Xander
Thanks, Filip,
nice idea.
Do you think if it would be good to integrate / build a Python module, which also will go through all GIS services and collect service statistics from ArcGIS Server REST 'logs' and export into Excel or SCV file?
Maybe, you have already this module. If so, please share...
++++++++++++++
The example is available at ArcGIS Help (10.2, 10.2.1, and 10.2.2)
it is an example with the chart in the report
Hi larry zhang, you seem to be able to use the code you referred to but I couldn't make it work.
What version of ArcGIS Server did you use? I thought the /arcgis/admin end point has been changed. Anyhow, I suppose general users don't have credentials for it anyway.
I tried to experiment with the following 10.2.x servers.
http://sampleserver5.arcgisonline.com/arcgis/rest/ (I have no credentials to this one but I couldn't even find the end points)
https://datahub.esriuk.com/arcgis/rest/ (I managed to generate token from https://datahub.esriuk.com/arcgis/tokens/generateToken but I cannot see any admin interface.
Here is the function I tried to get token with. Hopefully I'll figure out some way of using it to access secured services with arcapi/arrest.py.
import urllib2
import urllib
import json
from contextlib import closing
def generateToken(url, username, password):
params = urllib.urlencode({'username': username, 'password': password, 'client': 'requestip', 'f': 'json'})
headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}
req = urllib2.Request(url, params, headers)
with closing(urllib2.urlopen(req)) as res:
dta = res.read()
jsn = json.loads(dta)
return jsn.get('token')
username,password = '***', '***'
url = "https://datahub.esriuk.com/arcgis/tokens/generateToken"
token = generateToken(url, username, password)
Regards,
Filip.
Oh, thanks for the tip, Larry. The element of authentication makes it slightly more complex but I'll consider it if I can make a nice simple function out of it.
F.
Hi Filip,
ESRI in addition also produces their own Python API for accessing REST Services from ArcGIS Server and ArcGIS Online. The API for ArcGIS Server can be found here:
The API for ArcGIS Portal (ArcGIS Online) can be found here:
If you find something that ArcREST cannot do that arrest.py does, it may be beneficial to submit it to the ESRI repository. In addition, you may also want to consider a multitude of possible APIs to access REST services as I have found that one will support things that the others will not.
I hope this helps!
Thanks a lot, dear all, Greeting, Filip, pls refer to the blog at http://isbullsh.it/2012/06/Rest-api-in-python/ , which indicates that urllib2 is not right way to go... Hi, Alex, It looks that the links like https://github.com/esri/arcrest not working to me. Not sure why?
Filip,
Can you check the example codes to collect statistics of all services, which is available at ArcGIS Help (10.2, 10.2.1, and 10.2.2) ?
Hi Larry,
Thanks for the tip about urllib2. It is true that working with urllib2 is a bit awkward but exactly this awkwardness is removed if you use my arrest.py. More serious issues I've had with urllib2 were related to requesting data from https with self signed certificate. I would prefer the requests package but I didn't want to add a dependency on an external package. Is requests part of core Python in 3.x?
I also looked at the script to collect stats of all services you referred to and translated it into a function below. I could not test it because I don't have admin credentials to any server I can reach at the moment. Can you check if it works?
Filip.
"""Queries the logs and writes statistics on map service activity. Modified version of script available on ArcGIS Help: http://resources.arcgis.com/en/help/main/10.2/index.html#/Example_Derive_map_service_statistics_from... To try this example, follow these steps: 1) Publish several map services (without defining tile caches for them). 2) Set the ArcGIS Server log level to FINE. 3) Make some draw requests of your services by adding them to ArcMap etc. 4) Run this script and examine the resulting text file. """ import httplib, urllib, json import sys, time def collect_services_statistics(server_name, output_file, username, password, hours=24, port=6080): """Query ArcGIS Server logs and write a summary of all map service draws. server_name -- e.g. output_file -- path of the text file where the statistics should be written username, password -- credentials for the server admin rest interface hours=24 -- result will include summary for this number of the last hours port=6080 -- arcgis server port number """ def assertJsonSuccess(data): """Check that the input JSON object is not an error object.""" obj = json.loads(data) if 'status' in obj and obj['status'] == "error": print "Error: JSON object returns an error. " + str(obj) return False else: return True def getToken(username, password, server_name, port): """Generate a token given username, password and the adminURL.""" # Token URL is typically http://server[:port]/arcgis/admin/generateToken tokenURL = "/arcgis/admin/generateToken" # URL-encode the token parameters params = urllib.urlencode({'username': username, 'password': password, 'client': 'requestip', 'f': 'json'}) headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"} # Connect to URL and post parameters httpConn = httplib.HTTPConnection(server_name, port) httpConn.request("POST", tokenURL, params, headers) # Read response response = httpConn.getresponse() if (response.status != 200): httpConn.close() raise Exception("Unable to fetch token, check the URL and try again.") else: data = response.read() httpConn.close() # Check that data returned is not an error object if not assertJsonSuccess(data): return # Extract the toke from it token = json.loads(data) return token['token'] millisecondsToQuery = hours * 60 * 60 * 100 # 86400000 is 1 day hitDict = {} # Get a token token = getToken(username, password, server_name, port) if token == "": raise Exception("Could not generate a token using the credentials.") # Construct URL to query the logs logQueryURL = "/arcgis/admin/logs/query" # is this going to work in future? startTime = int(round(time.time() * 1000)) endTime = startTime - millisecondsToQuery logFilter = "{'services':'*','server':'*','machines':'*'}" params = urllib.urlencode({ 'level': 'FINE', 'startTime': startTime, 'endTime': endTime, 'filter':logFilter, 'token': token, 'f': 'json' }) headers = { "Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain" } # Connect to URL and post parameters httpConn = httplib.HTTPConnection(server_name, port) httpConn.request("POST", logQueryURL, params, headers) # Read response response = httpConn.getresponse() if (response.status != 200): httpConn.close() raise Exception("Error while querying logs.") else: data = response.read() # Check that data returned is not an error object if not assertJsonSuccess(data): raise Exception("Error returned by operation. " + data) # Deserialize response into Python object dataObj = json.loads(data) httpConn.close() # Need these to calculate average draw time for an ExportMapImage call mapDraws = 0 totalDrawTime = 0 # Iterate over messages for item in dataObj["logMessages"]: if item["message"] == "End ExportMapImage": elapsed = float(item["elapsed"]) keyCheck = item["source"] if keyCheck in hitDict: stats = hitDict[keyCheck] stats[0] += 1 # Add 1 to tally of hits stats[1] += elapsed # Add elapsed time to total elapsed time else: # Add key with one hit and total elapsed time hitDict[keyCheck] = [1, elapsed] # Open text file and write header line with open(output_file, "w") as summaryFile: header = "Service,Number of hits,Average seconds per draw\n" summaryFile.write(header) # Read through dictionary and write totals into file for key in hitDict: # Calculate average elapsed time totalDraws = hitDict[key][0] totalElapsed = hitDict[key][1] avgElapsed = 0 if totalDraws > 0: #Elapsed time divided by hits avgElapsed = (1.0 * (totalElapsed / totalDraws)) # Construct and write the comma-separated line line = key + "," + str(totalDraws) + "," + str(avgElapsed) + "\n" summaryFile.write(line) return summaryFile if __name__ == "__main__": server_name = 'http://sampleserver5.arcgisonline.com' output_file = 'c:/temp/summary.csv' username, password = "that_is_what", "i_do_not_know" collect_services_statistics(server_name, output_file, username, password, hours=24, port=6080)