Using a AGOL HFS in Python directly?

888
11
Jump to solution
04-17-2018 01:28 PM
DougBrowning
MVP Notable Contributor

In the past I have had some luck giving a AGOL HFS directly to a built in geoprocessing tool or script.  I usually created a layer file and fed that in.  BUT layer files refuse to save the tables - and most of what I want to use in tables.  In fact tables are always hard to get in ArcMap.  Try it - (Add a HFS with tables to ArcMap - save as layer file - open layer file - no tables! Or go to Add Data button - AGOL HFS - no tables! Or File - ArcOnline - Add - no tables!  Finally in Catalog go down to My Hosted Services - add the HFS - and actual tables!  But then save this out as a lyr file and tables are gone.).

What I would like to do is change my input source in Python from this

sourceDB = r"\\adgroup\blah\blah\data.gdb"

to something like

sourceDB = "direct link to HFS in AGOL"

Most of the script is Search Cursors so it would need to work with that.  This may be over hoping.

Any ideas on this? 

Backup plan is to use the python api to export/download a GDB (I already have this code) then unzip and feed to script.  But it would be nice to just go direct.

Another option may be ArcPro using v3?

Thanks for any ideas.

0 Kudos
1 Solution

Accepted Solutions
JonathanQuinn
Esri Frequent Contributor

What about querying the service, storing the data in a feature set, and then looping through the data as a JSON object?

>>> import json
>>> fs = arcpy.FeatureSet()
>>> fs.load("http://sampleserver6.arcgisonline.com/arcgis/rest/services/SampleWorldCities/MapServer/0/query?where...")
>>> fsJSON = json.loads(fs.JSON)
>>> fsJSON['features']
[{u'geometry': {u'y': -15.614990234374943, u'x': -56.09301757812494}, u'attributes': {u'OBJECTID': 1, u'POP_RANK': 3, u'POP': 521934, u'LABEL_FLAG': 0, u'CITY_NAME': u'Cuiaba', u'POP_CLASS': u'500,000 to 999,999'}}, {u'geometry': {u'y': -15.792114257812386, u'x': -47.897705078125}, u'attributes': {u'OBJECTID': 2, u'POP_RANK': 2, u'POP': 2207718, u'LABEL_FLAG': 0, u'CITY_NAME': u'Brasilia', u'POP_CLASS': u'1,000,000 to 4,999,999'}}, {u'geometry': {u'y': -16.726989746093693, u'x': -

You can then pull out any information you need to through loops or indexing:

>>> fsJSON['features'][0]
{u'geometry': {u'y': -15.614990234374943, u'x': -56.09301757812494}, u'attributes': {u'OBJECTID': 1, u'POP_RANK': 3, u'POP': 521934, u'LABEL_FLAG': 0, u'CITY_NAME': u'Cuiaba', u'POP_CLASS': u'500,000 to 999,999'}}

This will work with anything that supports querying.

View solution in original post

0 Kudos
11 Replies
JonathanQuinn
Esri Frequent Contributor

What about querying the service, storing the data in a feature set, and then looping through the data as a JSON object?

>>> import json
>>> fs = arcpy.FeatureSet()
>>> fs.load("http://sampleserver6.arcgisonline.com/arcgis/rest/services/SampleWorldCities/MapServer/0/query?where...")
>>> fsJSON = json.loads(fs.JSON)
>>> fsJSON['features']
[{u'geometry': {u'y': -15.614990234374943, u'x': -56.09301757812494}, u'attributes': {u'OBJECTID': 1, u'POP_RANK': 3, u'POP': 521934, u'LABEL_FLAG': 0, u'CITY_NAME': u'Cuiaba', u'POP_CLASS': u'500,000 to 999,999'}}, {u'geometry': {u'y': -15.792114257812386, u'x': -47.897705078125}, u'attributes': {u'OBJECTID': 2, u'POP_RANK': 2, u'POP': 2207718, u'LABEL_FLAG': 0, u'CITY_NAME': u'Brasilia', u'POP_CLASS': u'1,000,000 to 4,999,999'}}, {u'geometry': {u'y': -16.726989746093693, u'x': -

You can then pull out any information you need to through loops or indexing:

>>> fsJSON['features'][0]
{u'geometry': {u'y': -15.614990234374943, u'x': -56.09301757812494}, u'attributes': {u'OBJECTID': 1, u'POP_RANK': 3, u'POP': 521934, u'LABEL_FLAG': 0, u'CITY_NAME': u'Cuiaba', u'POP_CLASS': u'500,000 to 999,999'}}

This will work with anything that supports querying.

0 Kudos
DougBrowning
MVP Notable Contributor

Search Cursor would not work though right?  I was hoping to have one code stack that works for gdb, sde, or a HFS URL.

Thanks

0 Kudos
JonathanQuinn
Esri Frequent Contributor

Looks like it does!

>>> with arcpy.da.SearchCursor(fs,"*") as cursor:
...     for row in cursor:
...         print(row)
...         
(1, u'Cuiaba', 521934, 3, u'500,000 to 999,999', 0, (-56.09301757812494, -15.614990234374943))
(2, u'Brasilia', 2207718, 2, u'1,000,000 to 4,999,999', 0, (-47.897705078125, -15.792114257812386))
(3, u'Goiania', 1171195, 2, u'1,000,000 to 4,999,999', 0, (-49.25500488281244, -16.726989746093693))
(4, u'Campo Grande', 729151, 3, u'500,000 to 999,999', 0, (-54.61590576171875, -20.450988769531193))
(6, u'Salto del Guaira', 7385, 7, u'Less than 50,000', 0, (-54.28332519531244, -24.04998779296875))
(8, u'Encarnacion', 74983, 6, u'50,000 to 99,999', 0, (-55.85101318359375, -27.37701416015625))
DougBrowning
MVP Notable Contributor

Awsome!  I wrote it up then hit the next issue though - logging in.  I have some code the grabs our SAML logins to do something with AGO but I am not sure how to blend the two. 

I will let you know how it goes unless you have any ideas.

Thanks a lot

0 Kudos
JonathanQuinn
Esri Frequent Contributor

You'll need to get a token:

Get an access token | ArcGIS for Developers 

There's a sample for Python. I'm not sure about thee SAML part though, you'll likely need another library:

Python - Logging in to Site with SAML 2.0 - Stack Overflow 

0 Kudos
DougBrowning
MVP Notable Contributor

We have a guy here that wrote all the SAML stuff.  I worked with him on this one and he ended up updating his code to get it to work - but we did get it to work.

He has it on GitHub if you want to check it out.  https://github.com/DOI-BLM/requests-arcgis-auth

Simple code was

import os
from uuid import uuid4
import requests
import arcpy

import requests
from requests_arcgis_auth import ArcGISPortalSAMLAuth

# Setup Parameters & Authentication
url = r'https://yourorg.maps.arcgis.com/sharing/rest'  # change me
client_id = r'jkahfkashdf'   # change me
item_id_to_query = r'sampleitemid'  # change me
s = requests.session()
s.auth = ArcGISPortalSAMLAuth(client_id)

# Make a request to the item & obtain the feature service endpoint
response = s.post(url+"/content/items/{}?f=json".format(item_id_to_query))
fs_endpoint = response.json().get('url')
print ("Item ID {} feature service is hosted at {}".format(item_id_to_query,fs_endpoint))

# Query the end-point.
#   Keep in mind, this only works because the service is 'federated' with the portal.
#   This would not work for ANY service (including public/anonymous) as it will attach the token, and this token is only good for the specific portal we are working with
q_response = s.get(fs_endpoint + "/0/query?f=json&where=1=1&outFields=*&returnGeometry=true")
features = q_response.json().get('features')
print ("Found {} records on layer 0 of the fs endpoint".format(len(features)))

# Load to a 'feature set'
#   Keep in mind, this works because the 'token' is present in the HTTP GET URL.
#   May not be supported if it needs to be a POST or the parameters end up in the body of the request
fs = arcpy.FeatureSet()
fs.load(q_response.url)

with arcpy.da.SearchCursor(fs,"*") as cursor:
        for row in cursor:
            print(row)

I am not tested my full script yet that has several SearchCursors but I am hopeful.  Will report back.

Thanks again!

0 Kudos
DougBrowning
MVP Notable Contributor

Well I spoke too soon.  This is working for Feature Classes but it is giving an error on all tables.

This line gapLayer.load(q_response.url) gives this error

Traceback (most recent call last):
 line 226, in <module>
    gapLayer.load(q_response.url)
  File "C:\Program Files (x86)\ArcGIS\Desktop10.4\ArcPy\arcpy\arcobjects\arcobjects.py", line 175, in load
    return convertArcObjectToPythonObject(self._arc_object.Load(*gp_fixargs(args)))
RuntimeError: RecordSetObject: Cannot open table for Load

I tied changing gapLayer = arcpy.FeatureSet() to gapLayer = arcpy.RecordSet() but I get the same error. (I was guessing it was due to not being a FC).  I also tried taking out the returnGeometry=true part but still the same error.

The JSON all looks good just the load function is not happy.

Any ideas?

Thanks

0 Kudos
JonathanQuinn
Esri Frequent Contributor

I'd say using a RecordSet was a first good step, but looks like that's failing as well. Let me give this a try and see how it goes. You're using a table in a map service or feature service as input, correct?

0 Kudos
DougBrowning
MVP Notable Contributor

All layers and tables are from the same Feature Service.  The URL is working if I paste it in the browser just load will not take it.

Thanks!

0 Kudos