example for using token for exporting secured feature service to feature class

[tregchristopher1]

I'd like to use the ESRI blog example for exporting a feature service to a feature class using Python. This simple example is found at the bottom of:http://blogs.esri.com/esri/arcgis/2013/10/10/quick-tips-consuming-feature-services-with-geoprocessin...


I've been able to get this to work for non-secured services or for a secured service where I am already logged in to the REST endpoint through my browser. However, I can't figure out the format for replacing the empty token variable used in the example with my actual ArcGIS Server credentials that I would use if I viewed the feature service in a browser.

Thanks for any help you can provide.

[KevinHibma]

You can use this code to obtain a token....

import json, urllib, urllib2

USERNAME = "____"
PASSWORD = "_____"
REFER = "http://www.arcgis.com"

def urlopen(url, data=None):
    referer = "http://arcgis.com/arcgis/rest"
    req = urllib2.Request(url)
    req.add_header('Referer', referer)

    if data:
        response = urllib2.urlopen(req, data)
    else:
        response = urllib2.urlopen(req)

    return response


def gentoken(username, password, referer, expiration=60):
    #Re-usable function to get a token   
    
    query_dict = {'username': username,
                  'password': password,
                  'expiration': str(expiration),
                  'client': 'referer',
                  'referer': referer,
                  'f': 'json'}    
    
    query_string = urllib.urlencode(query_dict)
    tokenUrl = "https://www.arcgis.com/sharing/rest/generateToken"
    
    tokenResponse = urllib.urlopen(tokenUrl, urllib.urlencode(query_dict))   
    token = json.loads(tokenResponse.read())
        
    if "token" not in token:
        print token['messages']
        exit()
    else:
        # Return the token to the function which called for it
        return token['token']
    
    
token = gentoken(USERNAME, PASSWORD, REFER)

[tregchristopher1]

Thanks for the code Kevin. Unfortunately, I'm still having problems.

When I use your token code, the token is being generated but I still can't access the feature service. If I message the URL + Query I get the message below showing the token (URL has been partially rewritten):
"https://FishAndWildlifeURL.gov/arcgis/rest/services/Wildlife/WildlifePtObs/FeatureServer/0/query?whe..."

But then I get the following errors:
Traceback (most recent call last):
  File "C:\data\Projects\WildlifeObs_DataExportTool.py", line 72, in <module>
    fs.load(fsURL)
  File "c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\arcobjects\arcobjects.py", line 173, in load
    return convertArcObjectToPythonObject(self._arc_object.Load(*gp_fixargs(args)))
RuntimeError: RecordSetObject: Cannot open table for Load

There are a few things that are different about our configuration that may be the problem?:
-it's a https address instead of http
-there may be a problem with the web adapter because I have to use a port# in the tokenurl (tokenUrl = "https://FishAndWildlifeURL.gov:6443/arcgis/tokens/")
-the machine I'm running the script on is behind a firewall that is different from where the feature service is stored

Any ideas? Keep in mind that if I set the token to blank and log into the secured REST endpoint in my web browser I can run the code and get a feature class produced, so it still a problem with security and not the conversion of feat service to feat class.

[tregchristopher1]

OK, I finally got a solution to work (posted below). The code is different from what Kevin posted above but the key to making this work was to change the value for the SERVER variable (or REFER in Kevin's example) from something like "https://myGovWebsite.gov" to just "myGovWebsite.gov" (no http prefix). Sorry about the missing indents for the Python code below.

import arcpy, json, urllib, httplib

USERNAME = "____"
PASSWORD = "_____"
SERVERNAME = "FishAndWildlifeURL.gov"#omit the "https://" prefix for this to work
PORT = "6443"

def getToken(username, password, serverName, serverPort):
tokenURL = "https://FishAndWildlifeURL:6443/arcgis/admin/generateToken"

params = urllib.urlencode({'username': username, 'password': password,'client': 'requestip', 'f': 'json'})

headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}

# Connect to URL and post parameters
httpConn = httplib.HTTPSConnection(serverName, serverPort)
httpConn.request("POST", tokenURL, params, headers)

# Read response
response = httpConn.getresponse()
print response.status
if (response.status != 200):
  httpConn.close()
  print "Error while fetching tokens from admin URL. Please check the URL and try again."
  return
else:
data = response.read()
httpConn.close()

# Check that data returned is not an error object
if not assertJsonSuccess(data):
  return

# Extract the token from it
token = json.loads(data)
return token['token']

###End getToken function

# A function that checks that the input JSON object is not an error object.
def assertJsonSuccess(data):
obj = json.loads(data)
if 'status' in obj and obj['status'] == "error":
  print "Error: JSON object returns an error. " + str(obj)
  return False
else:
  return True

### End jsonSuccess function


#Query for service, with URL, where clause, fields and token
baseURL= "https://FishAndWildlifeURL:6443/arcgis/rest/services/myFolder/myObs/FeatureServer/0/query"
where = '1=1'
fields = '*'
token = getToken(USERNAME, PASSWORD, SERVERNAME, PORT)
query = "?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, fields, token)
fsURL = baseURL + query
fs = arcpy.FeatureSet()
fs.load(fsURL)

arcpy.CopyFeatures_management(fs, r"C:\Default.gdb\temp_obs")

[MatthewWalker]

Excellent work!  This is just what I needed!

Some modifications I had to make to Treg's code in order to get this to work, since it seems Python 3.x handles some things differently:

I had to import http.client instead of httplib

I had to use 'params = urllib.parse.urlencode(...' instead of 'params = urllib.urlencode(...'

I am using a web adaptor, so I left the PORT null, and just changed the tokenURL

Other than that, this worked!  Now I just need to figure out how to push edits/additions back up to the feature service via arcpy...