topic Re: Accessing REST API Query with username and password in Python Questions

Accessing REST API Query with username and password

Zeke — Fri, 22 Dec 2017 16:37:24 GMT

I have the following code that I want to use to access a secured arc rest api site and query a layer to return the data in json. I can do this manually by going to the rest site in a browser, signing in, and using the query page to return data. And the code below (mostly copied from the web) works on non secured sites, without username and password.

But when I try to access the site using the password, I get the error:

RuntimeError: RecordSetObject: Cannot open table for Load

How can I do log in programmatically to get this data? Thanks.

    baseURL = "http://company/arcgis/rest/services/Maps/CityMapServer/6/query"
    where = "1=1"
    fields = "*"
    user = "user"
    password = "pass"
    query = "?where={}&outFields={}&returnGeometry=true&f=json&auth=({},{})".format(where, fields, user, password)
    fsURL = baseURL + query
    fs = arcpy.FeatureSet()
    fs.load(fsURL)

Re: Accessing REST API Query with username and password

MicahBabinski — Fri, 22 Dec 2017 17:25:24 GMT

Hello Greg,

Do you know what manner of authentication the server you're trying to access uses? Frequently, access through the REST API is achieved using tokens. See Acquiring ArcGIS tokens for more info on that. For the API syntax, see Generate Token. Then I believe you would use this token response in JSON format in whatever request you are making to the secured service.

Micah

Re: Accessing REST API Query with username and password

RandyBurton — Sat, 11 Dec 2021 16:59:18 GMT

The procedure is to log on with a username and password to get a token. Then you add the token to your query, not the username/password. I use the following for AGOL. Server/Portal is slightly different - mostly in the URLs used; I would suggest looking at Jake Skinner‌'s Show Attachments in Web Map Popup as his script covers both versions.

import urllib
import urllib2
import json

# Credentials and feature service information
username = "username"
password = "password"

URL = "https://services2.arcgis.com/abc123/arcgis/rest/services/"

# obtain a token
referer = "http://www.arcgis.com/"
query_dict = { 'username': username, 'password': password, 'referer': referer }

query_string = urllib.urlencode(query_dict)
url = "https://www.arcgis.com/sharing/rest/generateToken"
token = json.loads(urllib.urlopen(url + "?f=json", query_string).read())

if "token" not in token:
    print(token['error'])
    sys.exit(1)

query_dict = { "f": "json", "token": token['token'] }
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Accessing REST API Query with username and password

JonathanQuinn — Sat, 11 Dec 2021 16:59:20 GMT

I use a couple functions to get a token and open URLs:

import json, urllib2, urllib, traceback, ssl

def generateToken(user, password):
    serverName = baseURL.split("/")[2]
    tokenURL = '{0}/tokens/generateToken'.format(baseURL)
    tokenParams = dict(username=user,password=password,client='referer',referer='https://{0}'.format(serverName),expiration=20160)
    tokenResponse = openURL(tokenURL,tokenParams)
    if isinstance(tokenResponse,dict):
        if "token" in tokenResponse:
            return tokenResponse['token']
        else:
            raise Exception("Unable to generate token.\n{0}".format(tokenResponse))
    else:
        raise Exception("Unable to generate token.\n{0}".format(tokenResponse))

def openURL(url,params=None, protocol=None):
    serverName = url.split("/")[2]
    try:
        if params:
            params.update(dict(f="json"))
        else:
            params = dict(f="json")
        if protocol:
            encoded_params = str.encode(urllib.urlencode(params))
            encoded_params = encoded_params.decode("utf-8")
            url = "{0}?{1}".format(url, encoded_params)
            request = urllib2.Request(url)
            sslContext = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
            request.add_header('referer','https://{0}'.format(serverName))
            response = urllib2.urlopen(request, context=sslContext)
        else:
            encoded_params = str.encode(urllib.urlencode(params))
            request = urllib2.Request(url)
            sslContext = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
            request.add_header('referer','https://{0}'.format(serverName))
            response = urllib2.urlopen(request,encoded_params,context=sslContext)
        decodedResponse = response.read().decode('utf-8')
        jsonResponse = json.loads(decodedResponse)
        return jsonResponse
    except urllib2.HTTPError as e:
        print(url)
        print(e)
        return e
    except urllib2.URLError as e:
        print(url)
        print(e)
        return e
    except:
        print(traceback.format_exc())‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So then you can do something like:

baseURL = 'https://sampleserver6.arcgisonline.com/arcgis'

token = generateToken('user1','user1')

queryParams = dict(where="1=1",token=token)
queryURL = 'https://sampleserver6.arcgisonline.com/arcgis/rest/services/Wildfire_secure/MapServer/0/query'

response = openURL(queryURL,queryParams)
print(response)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Accessing REST API Query with username and password

Zeke — Fri, 22 Dec 2017 20:49:23 GMT

Hmm, this is more complicated than I thought.

Micah, I don't know what they use. Rest endpoint has Sign in link, and Generate token link. I've just signed in.

Even the manual approach got me kudos at work, but you know how that turns into "Cool. Now do more". That's ok, enjoy the challenge.

Thanks all, I'll try these approaches out and report on results.

Re: Accessing REST API Query with username and password

MicahBabinski — Fri, 22 Dec 2017 22:33:59 GMT

Very cool. The code samples provided look similar to what I've worked with

in the past. You are on the right track scaling up what you can deliver

with the rest API.

Re: Accessing REST API Query with username and password

Zeke — Fri, 29 Dec 2017 13:03:27 GMT

Ok, so I used the code above along with the requests module and got it almost all working (probably don't have to use both urllib and requests, but it works). Works fine, except for one thing: the REST data is in WGS_1984_Web_Mercator_Auxiliary_Sphere, and I want to import it into WGS 1984. From what I've read, setting outSR to 4326 should do it, but no luck. Still imports in Web Mercator. If I go to the rest endpoint in a browser and set it, works correctly, just not in the script. Is this a bug, am I setting it wrong, ??? Thanks.(Sorry, forgotten how to format as code).

def getJson(token):
    # query parameters
    query_dict = {"token" : token["token"], "where" : "1=1", "outfields" : "*", "outSR" : "4326", "f" : "json"}
    baseurl = "http://company/arcgis/rest/services/Maps/city/MapServer/"

    # layers in REST endpoints are identified by index number, not name
    layerid = layer_dict[infeature]
    query = "/query"
    urltoquery = baseurl + layerid + query

    r = requests.get(urltoquery, auth=(username, password), params=query_dict)

    # build json file name, saved to user desktop
    currentuser = os.getenv('username')
    ext = ".json"
    filenm = os.path.join(r"C:\Users", currentuser, r"Desktop\project" + infeature) + ext

    with open(filenm, "wb") as f:
        for line in r.iter_content():
            f.write(line)
    try:
        arcpy.JSONToFeatures_conversion(filenm, outfeature)
    except Exception as e:
        err = "\nError importing json file\n{}".format(e)
        arcpy.AddError(err)

Re: Accessing REST API Query with username and password

MeghanKulkarni — Wed, 02 Jan 2019 16:17:09 GMT

Hello Jonathan,

Can help with what's str in function openURL()? this line to be precise?

  encoded_params = str.encode(urllib.urlencode(params))

Re: Accessing REST API Query with username and password

JonathanQuinn — Thu, 03 Jan 2019 17:17:19 GMT

That's necessary for Python 3 to convert Unicode to strings.

Python String encode() - Python Standard Library

Re: Accessing REST API Query with username and password

MeghanKulkarni — Fri, 04 Jan 2019 20:23:29 GMT

Thanks Jonathan

Re: Accessing REST API Query with username and password

JamesGraham — Fri, 22 May 2020 19:29:46 GMT

Just wanted to add this for future generations who are still working with ArcGIS Server-based Feature Services and want to use arcpy/Python to authenticate via token to do stuff.

I could not get any of the above to work in my environments (10.8, 10.6.1), but found an outstanding Python toolset called 'restapi' that is great if you still need to run things on Python 2.7.x and more traditional ArcGIS Servers (that are not Portal).

Authenticating is easy (token)
Editing is easy (including adding/editing attachments for feature services)

Highly recommend this repository!

GitHub - Bolton-and-Menk-GIS/restapi: Python API designed to work externally with ArcGIS REST Services to query and extr…