Using a AGOL HFS in Python directly?

[DougBrowning]

In the past I have had some luck giving a AGOL HFS directly to a built in geoprocessing tool or script.  I usually created a layer file and fed that in.  BUT layer files refuse to save the tables - and most of what I want to use in tables.  In fact tables are always hard to get in ArcMap.  Try it - (Add a HFS with tables to ArcMap - save as layer file - open layer file - no tables! Or go to Add Data button - AGOL HFS - no tables! Or File - ArcOnline - Add - no tables!  Finally in Catalog go down to My Hosted Services - add the HFS - and actual tables!  But then save this out as a lyr file and tables are gone.).

What I would like to do is change my input source in Python from this

sourceDB = r"\\adgroup\blah\blah\data.gdb"

to something like

sourceDB = "direct link to HFS in AGOL"

Most of the script is Search Cursors so it would need to work with that.  This may be over hoping.

Any ideas on this? 

Backup plan is to use the python api to export/download a GDB (I already have this code) then unzip and feed to script.  But it would be nice to just go direct.

Another option may be ArcPro using v3?

Thanks for any ideas.

[JonathanQuinn · Accepted solution]

What about querying the service, storing the data in a feature set, and then looping through the data as a JSON object?

>>> import json
>>> fs = arcpy.FeatureSet()
>>> fs.load("http://sampleserver6.arcgisonline.com/arcgis/rest/services/SampleWorldCities/MapServer/0/query?where...")
>>> fsJSON = json.loads(fs.JSON)
>>> fsJSON['features']
[{u'geometry': {u'y': -15.614990234374943, u'x': -56.09301757812494}, u'attributes': {u'OBJECTID': 1, u'POP_RANK': 3, u'POP': 521934, u'LABEL_FLAG': 0, u'CITY_NAME': u'Cuiaba', u'POP_CLASS': u'500,000 to 999,999'}}, {u'geometry': {u'y': -15.792114257812386, u'x': -47.897705078125}, u'attributes': {u'OBJECTID': 2, u'POP_RANK': 2, u'POP': 2207718, u'LABEL_FLAG': 0, u'CITY_NAME': u'Brasilia', u'POP_CLASS': u'1,000,000 to 4,999,999'}}, {u'geometry': {u'y': -16.726989746093693, u'x': -‍‍‍‍‍‍

You can then pull out any information you need to through loops or indexing:

>>> fsJSON['features'][0]
{u'geometry': {u'y': -15.614990234374943, u'x': -56.09301757812494}, u'attributes': {u'OBJECTID': 1, u'POP_RANK': 3, u'POP': 521934, u'LABEL_FLAG': 0, u'CITY_NAME': u'Cuiaba', u'POP_CLASS': u'500,000 to 999,999'}}‍‍

This will work with anything that supports querying.

[DougBrowning]

Search Cursor would not work though right?  I was hoping to have one code stack that works for gdb, sde, or a HFS URL.

Thanks

[JonathanQuinn]

Looks like it does!

>>> with arcpy.da.SearchCursor(fs,"*") as cursor:
...     for row in cursor:
...         print(row)
...         
(1, u'Cuiaba', 521934, 3, u'500,000 to 999,999', 0, (-56.09301757812494, -15.614990234374943))
(2, u'Brasilia', 2207718, 2, u'1,000,000 to 4,999,999', 0, (-47.897705078125, -15.792114257812386))
(3, u'Goiania', 1171195, 2, u'1,000,000 to 4,999,999', 0, (-49.25500488281244, -16.726989746093693))
(4, u'Campo Grande', 729151, 3, u'500,000 to 999,999', 0, (-54.61590576171875, -20.450988769531193))
(6, u'Salto del Guaira', 7385, 7, u'Less than 50,000', 0, (-54.28332519531244, -24.04998779296875))
(8, u'Encarnacion', 74983, 6, u'50,000 to 99,999', 0, (-55.85101318359375, -27.37701416015625))
‍‍‍‍‍‍‍‍‍‍

[DougBrowning]

Awsome!  I wrote it up then hit the next issue though - logging in.  I have some code the grabs our SAML logins to do something with AGO but I am not sure how to blend the two. 

I will let you know how it goes unless you have any ideas.

Thanks a lot

[JonathanQuinn]

You'll need to get a token:

Get an access token | ArcGIS for Developers 

There's a sample for Python. I'm not sure about thee SAML part though, you'll likely need another library:

Python - Logging in to Site with SAML 2.0 - Stack Overflow 

[DougBrowning]

We have a guy here that wrote all the SAML stuff.  I worked with him on this one and he ended up updating his code to get it to work - but we did get it to work.

He has it on GitHub if you want to check it out.  https://github.com/DOI-BLM/requests-arcgis-auth

Simple code was

import os
from uuid import uuid4
import requests
import arcpy

import requests
from requests_arcgis_auth import ArcGISPortalSAMLAuth

# Setup Parameters & Authentication
url = r'https://yourorg.maps.arcgis.com/sharing/rest'  # change me
client_id = r'jkahfkashdf'   # change me
item_id_to_query = r'sampleitemid'  # change me
s = requests.session()
s.auth = ArcGISPortalSAMLAuth(client_id)

# Make a request to the item & obtain the feature service endpoint
response = s.post(url+"/content/items/{}?f=json".format(item_id_to_query))
fs_endpoint = response.json().get('url')
print ("Item ID {} feature service is hosted at {}".format(item_id_to_query,fs_endpoint))

# Query the end-point.
#   Keep in mind, this only works because the service is 'federated' with the portal.
#   This would not work for ANY service (including public/anonymous) as it will attach the token, and this token is only good for the specific portal we are working with
q_response = s.get(fs_endpoint + "/0/query?f=json&where=1=1&outFields=*&returnGeometry=true")
features = q_response.json().get('features')
print ("Found {} records on layer 0 of the fs endpoint".format(len(features)))

# Load to a 'feature set'
#   Keep in mind, this works because the 'token' is present in the HTTP GET URL.
#   May not be supported if it needs to be a POST or the parameters end up in the body of the request
fs = arcpy.FeatureSet()
fs.load(q_response.url)

with arcpy.da.SearchCursor(fs,"*") as cursor:
        for row in cursor:
            print(row)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I am not tested my full script yet that has several SearchCursors but I am hopeful.  Will report back.

Thanks again!

[DougBrowning]

Well I spoke too soon.  This is working for Feature Classes but it is giving an error on all tables.

This line gapLayer.load(q_response.url) gives this error

Traceback (most recent call last):
 line 226, in <module>
    gapLayer.load(q_response.url)
  File "C:\Program Files (x86)\ArcGIS\Desktop10.4\ArcPy\arcpy\arcobjects\arcobjects.py", line 175, in load
    return convertArcObjectToPythonObject(self._arc_object.Load(*gp_fixargs(args)))
RuntimeError: RecordSetObject: Cannot open table for Load

I tied changing gapLayer = arcpy.FeatureSet() to gapLayer = arcpy.RecordSet() but I get the same error. (I was guessing it was due to not being a FC).  I also tried taking out the returnGeometry=true part but still the same error.

The JSON all looks good just the load function is not happy.

Any ideas?

Thanks

[JonathanQuinn]

I'd say using a RecordSet was a first good step, but looks like that's failing as well. Let me give this a try and see how it goes. You're using a table in a map service or feature service as input, correct?

[DougBrowning]

All layers and tables are from the same Feature Service.  The URL is working if I paste it in the browser just load will not take it.

Thanks!