Accessing REST API Query with username and password

[Zeke]

I have the following code that I want to use to access a secured arc rest api site and query a layer to return the data in json. I can do this manually by going to the rest site in a browser, signing in, and using the query page to return data. And the code below (mostly copied from the web) works on non secured sites, without username and password.

But when I try to access the site using the password, I get the error:

RuntimeError: RecordSetObject: Cannot open table for Load  

How can I do log in programmatically to get this data? Thanks.

    baseURL = "http://company/arcgis/rest/services/Maps/CityMapServer/6/query"
    where = "1=1"
    fields = "*"
    user = "user"
    password = "pass"
    query = "?where={}&outFields={}&returnGeometry=true&f=json&auth=({},{})".format(where, fields, user, password)
    fsURL = baseURL + query
    fs = arcpy.FeatureSet()
    fs.load(fsURL)

[MicahBabinski]

Hello Greg,

Do you know what manner of authentication the server you're trying to access uses? Frequently, access through the REST API is achieved using tokens. See Acquiring ArcGIS tokens for more info on that. For the API syntax, see Generate Token. Then I believe you would use this token response in JSON format in whatever request you are making to the secured service.

Micah

[RandyBurton · Accepted solution]

The procedure is to log on with a username and password to get a token.  Then you add the token to your query, not the username/password.  I use the following for AGOL.  Server/Portal is slightly different - mostly in the URLs used; I would suggest looking at Jake Skinner‌'s Show Attachments in Web Map Popup as his script covers both versions.

import urllib
import urllib2
import json

# Credentials and feature service information
username = "username"
password = "password"

URL = "https://services2.arcgis.com/abc123/arcgis/rest/services/"

# obtain a token
referer = "http://www.arcgis.com/"
query_dict = { 'username': username, 'password': password, 'referer': referer }

query_string = urllib.urlencode(query_dict)
url = "https://www.arcgis.com/sharing/rest/generateToken"
token = json.loads(urllib.urlopen(url + "?f=json", query_string).read())

if "token" not in token:
    print(token['error'])
    sys.exit(1)

query_dict = { "f": "json", "token": token['token'] }
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[JonathanQuinn]

I use a couple functions to get a token and open URLs:

import json, urllib2, urllib, traceback, ssl

def generateToken(user, password):
    serverName = baseURL.split("/")[2]
    tokenURL = '{0}/tokens/generateToken'.format(baseURL)
    tokenParams = dict(username=user,password=password,client='referer',referer='https://{0}'.format(serverName),expiration=20160)
    tokenResponse = openURL(tokenURL,tokenParams)
    if isinstance(tokenResponse,dict):
        if "token" in tokenResponse:
            return tokenResponse['token']
        else:
            raise Exception("Unable to generate token.\n{0}".format(tokenResponse))
    else:
        raise Exception("Unable to generate token.\n{0}".format(tokenResponse))

def openURL(url,params=None, protocol=None):
    serverName = url.split("/")[2]
    try:
        if params:
            params.update(dict(f="json"))
        else:
            params = dict(f="json")
        if protocol:
            encoded_params = str.encode(urllib.urlencode(params))
            encoded_params = encoded_params.decode("utf-8")
            url = "{0}?{1}".format(url, encoded_params)
            request = urllib2.Request(url)
            sslContext = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
            request.add_header('referer','https://{0}'.format(serverName))
            response = urllib2.urlopen(request, context=sslContext)
        else:
            encoded_params = str.encode(urllib.urlencode(params))
            request = urllib2.Request(url)
            sslContext = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
            request.add_header('referer','https://{0}'.format(serverName))
            response = urllib2.urlopen(request,encoded_params,context=sslContext)
        decodedResponse = response.read().decode('utf-8')
        jsonResponse = json.loads(decodedResponse)
        return jsonResponse
    except urllib2.HTTPError as e:
        print(url)
        print(e)
        return e
    except urllib2.URLError as e:
        print(url)
        print(e)
        return e
    except:
        print(traceback.format_exc())‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So then you can do something like:

baseURL = 'https://sampleserver6.arcgisonline.com/arcgis'

token = generateToken('user1','user1')

queryParams = dict(where="1=1",token=token)
queryURL = 'https://sampleserver6.arcgisonline.com/arcgis/rest/services/Wildfire_secure/MapServer/0/query'

response = openURL(queryURL,queryParams)
print(response)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[MeghanKulkarni]

Hello Jonathan,

Can help with what's str in function openURL()? this line to be precise? 

  encoded_params = str.encode(urllib.urlencode(params))

[JonathanQuinn]

That's necessary for Python 3 to convert Unicode to strings.

Python String encode() - Python Standard Library 

[MeghanKulkarni]

Thanks Jonathan

[Zeke]

Hmm, this is more complicated than I thought.

Micah, I don't know what they use. Rest endpoint has Sign in link, and Generate token link. I've just signed in.

Even the manual approach got me kudos at work, but you know how that turns into "Cool. Now do more". That's ok, enjoy the challenge.

Thanks all, I'll try these approaches out and report on results.

[MicahBabinski]

Very cool. The code samples provided look similar to what I've worked with

in the past. You are on the right track scaling up what you can deliver

with the rest API.

[Zeke]

Ok, so I used the code above along with the requests module and got it almost all working (probably don't have to use both urllib and requests, but it works). Works fine, except for one thing: the REST data is in WGS_1984_Web_Mercator_Auxiliary_Sphere, and I want to import it into WGS 1984. From what I've read, setting outSR to 4326 should do it, but no luck. Still imports in Web Mercator. If I go to the rest endpoint in a browser and set it, works correctly, just not in the script. Is this a bug, am I setting it wrong, ??? Thanks.(Sorry, forgotten how to format as code).

def getJson(token):
    # query parameters
    query_dict = {"token" : token["token"], "where" : "1=1", "outfields" : "*", "outSR" : "4326", "f" : "json"}
    baseurl = "http://company/arcgis/rest/services/Maps/city/MapServer/"

    # layers in REST endpoints are identified by index number, not name
    layerid = layer_dict[infeature]
    query = "/query"
    urltoquery = baseurl + layerid + query

    r = requests.get(urltoquery, auth=(username, password), params=query_dict)

    # build json file name, saved to user desktop
    currentuser = os.getenv('username')
    ext = ".json"
    filenm = os.path.join(r"C:\Users", currentuser, r"Desktop\project" + infeature) + ext

    with open(filenm, "wb") as f:
        for line in r.iter_content():
            f.write(line)
    try:
        arcpy.JSONToFeatures_conversion(filenm, outfeature)
    except Exception as e:
        err = "\nError importing json file\n{}".format(e)
        arcpy.AddError(err)