topic Re: Using a AGOL HFS in Python directly? in Python Questions

Using a AGOL HFS in Python directly?

DougBrowning — Tue, 17 Apr 2018 20:28:36 GMT

In the past I have had some luck giving a AGOL HFS directly to a built in geoprocessing tool or script. I usually created a layer file and fed that in. BUT layer files refuse to save the tables - and most of what I want to use in tables. In fact tables are always hard to get in ArcMap. Try it - (Add a HFS with tables to ArcMap - save as layer file - open layer file - no tables! Or go to Add Data button - AGOL HFS - no tables! Or File - ArcOnline - Add - no tables! Finally in Catalog go down to My Hosted Services - add the HFS - and actual tables! But then save this out as a lyr file and tables are gone.).

What I would like to do is change my input source in Python from this

sourceDB = r"\\adgroup\blah\blah\data.gdb"

to something like

sourceDB = "direct link to HFS in AGOL"

Most of the script is Search Cursors so it would need to work with that. This may be over hoping.

Any ideas on this?

Backup plan is to use the python api to export/download a GDB (I already have this code) then unzip and feed to script. But it would be nice to just go direct.

Another option may be ArcPro using v3?

Thanks for any ideas.

Re: Using a AGOL HFS in Python directly?

JonathanQuinn — Fri, 10 Dec 2021 23:17:50 GMT

What about querying the service, storing the data in a feature set, and then looping through the data as a JSON object?

>>> import json
>>> fs = arcpy.FeatureSet()
>>> fs.load("http://sampleserver6.arcgisonline.com/arcgis/rest/services/SampleWorldCities/MapServer/0/query?where=1%3D1&outFields=*&returnGeometry=true&f=json")
>>> fsJSON = json.loads(fs.JSON)
>>> fsJSON['features']
[{u'geometry': {u'y': -15.614990234374943, u'x': -56.09301757812494}, u'attributes': {u'OBJECTID': 1, u'POP_RANK': 3, u'POP': 521934, u'LABEL_FLAG': 0, u'CITY_NAME': u'Cuiaba', u'POP_CLASS': u'500,000 to 999,999'}}, {u'geometry': {u'y': -15.792114257812386, u'x': -47.897705078125}, u'attributes': {u'OBJECTID': 2, u'POP_RANK': 2, u'POP': 2207718, u'LABEL_FLAG': 0, u'CITY_NAME': u'Brasilia', u'POP_CLASS': u'1,000,000 to 4,999,999'}}, {u'geometry': {u'y': -16.726989746093693, u'x': -‍‍‍‍‍‍

You can then pull out any information you need to through loops or indexing:

>>> fsJSON['features'][0]
{u'geometry': {u'y': -15.614990234374943, u'x': -56.09301757812494}, u'attributes': {u'OBJECTID': 1, u'POP_RANK': 3, u'POP': 521934, u'LABEL_FLAG': 0, u'CITY_NAME': u'Cuiaba', u'POP_CLASS': u'500,000 to 999,999'}}‍‍

This will work with anything that supports querying.

Re: Using a AGOL HFS in Python directly?

DougBrowning — Thu, 19 Apr 2018 16:22:45 GMT

Search Cursor would not work though right? I was hoping to have one code stack that works for gdb, sde, or a HFS URL.

Thanks

Re: Using a AGOL HFS in Python directly?

JonathanQuinn — Fri, 10 Dec 2021 23:17:52 GMT

Looks like it does!

>>> with arcpy.da.SearchCursor(fs,"*") as cursor:
...     for row in cursor:
...         print(row)
...         
(1, u'Cuiaba', 521934, 3, u'500,000 to 999,999', 0, (-56.09301757812494, -15.614990234374943))
(2, u'Brasilia', 2207718, 2, u'1,000,000 to 4,999,999', 0, (-47.897705078125, -15.792114257812386))
(3, u'Goiania', 1171195, 2, u'1,000,000 to 4,999,999', 0, (-49.25500488281244, -16.726989746093693))
(4, u'Campo Grande', 729151, 3, u'500,000 to 999,999', 0, (-54.61590576171875, -20.450988769531193))
(6, u'Salto del Guaira', 7385, 7, u'Less than 50,000', 0, (-54.28332519531244, -24.04998779296875))
(8, u'Encarnacion', 74983, 6, u'50,000 to 99,999', 0, (-55.85101318359375, -27.37701416015625))
‍‍‍‍‍‍‍‍‍‍

Re: Using a AGOL HFS in Python directly?

DougBrowning — Thu, 19 Apr 2018 21:56:28 GMT

Awsome! I wrote it up then hit the next issue though - logging in. I have some code the grabs our SAML logins to do something with AGO but I am not sure how to blend the two.

I will let you know how it goes unless you have any ideas.

Thanks a lot

Re: Using a AGOL HFS in Python directly?

JonathanQuinn — Thu, 19 Apr 2018 22:00:05 GMT

You'll need to get a token:

Get an access token | ArcGIS for Developers

There's a sample for Python. I'm not sure about thee SAML part though, you'll likely need another library:

Python - Logging in to Site with SAML 2.0 - Stack Overflow

Re: Using a AGOL HFS in Python directly?

DougBrowning — Fri, 10 Dec 2021 23:17:55 GMT

We have a guy here that wrote all the SAML stuff. I worked with him on this one and he ended up updating his code to get it to work - but we did get it to work.

He has it on GitHub if you want to check it out. https://github.com/DOI-BLM/requests-arcgis-auth

Simple code was

import os
from uuid import uuid4
import requests
import arcpy

import requests
from requests_arcgis_auth import ArcGISPortalSAMLAuth

# Setup Parameters & Authentication
url = r'https://yourorg.maps.arcgis.com/sharing/rest'  # change me
client_id = r'jkahfkashdf'   # change me
item_id_to_query = r'sampleitemid'  # change me
s = requests.session()
s.auth = ArcGISPortalSAMLAuth(client_id)

# Make a request to the item & obtain the feature service endpoint
response = s.post(url+"/content/items/{}?f=json".format(item_id_to_query))
fs_endpoint = response.json().get('url')
print ("Item ID {} feature service is hosted at {}".format(item_id_to_query,fs_endpoint))

# Query the end-point.
#   Keep in mind, this only works because the service is 'federated' with the portal.
#   This would not work for ANY service (including public/anonymous) as it will attach the token, and this token is only good for the specific portal we are working with
q_response = s.get(fs_endpoint + "/0/query?f=json&where=1=1&outFields=*&returnGeometry=true")
features = q_response.json().get('features')
print ("Found {} records on layer 0 of the fs endpoint".format(len(features)))

# Load to a 'feature set'
#   Keep in mind, this works because the 'token' is present in the HTTP GET URL.
#   May not be supported if it needs to be a POST or the parameters end up in the body of the request
fs = arcpy.FeatureSet()
fs.load(q_response.url)

with arcpy.da.SearchCursor(fs,"*") as cursor:
        for row in cursor:
            print(row)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I am not tested my full script yet that has several SearchCursors but I am hopeful. Will report back.

Thanks again!

Re: Using a AGOL HFS in Python directly?

DougBrowning — Fri, 20 Apr 2018 19:52:28 GMT

Well I spoke too soon. This is working for Feature Classes but it is giving an error on all tables.

This line gapLayer.load(q_response.url) gives this error

Traceback (most recent call last):
line 226, in <module>
gapLayer.load(q_response.url)
File "C:\Program Files (x86)\ArcGIS\Desktop10.4\ArcPy\arcpy\arcobjects\arcobjects.py", line 175, in load
return convertArcObjectToPythonObject(self._arc_object.Load(*gp_fixargs(args)))
RuntimeError: RecordSetObject: Cannot open table for Load

I tied changing gapLayer = arcpy.FeatureSet() to gapLayer = arcpy.RecordSet() but I get the same error. (I was guessing it was due to not being a FC). I also tried taking out the returnGeometry=true part but still the same error.

The JSON all looks good just the load function is not happy.

Any ideas?

Thanks

Re: Using a AGOL HFS in Python directly?

JonathanQuinn — Fri, 20 Apr 2018 20:14:28 GMT

I'd say using a RecordSet was a first good step, but looks like that's failing as well. Let me give this a try and see how it goes. You're using a table in a map service or feature service as input, correct?

Re: Using a AGOL HFS in Python directly?

DougBrowning — Fri, 20 Apr 2018 20:22:43 GMT

All layers and tables are from the same Feature Service. The URL is working if I paste it in the browser just load will not take it.

Thanks!

Re: Using a AGOL HFS in Python directly?

JonathanQuinn — Fri, 10 Dec 2021 23:17:58 GMT

What does your script look like? This works for me:

import json, arcpy
fs = arcpy.FeatureSet()
fs.load("http://sampleserver6.arcgisonline.com/arcgis/rest/services/ServiceRequest/FeatureServer/0/query?where=1%3D1&outFields=*&returnGeometry=true&f=json")
fsJSON = json.loads(fs.JSON)
with arcpy.da.SearchCursor(fs,"*") as cursor:
    for row in cursor:
        print(row)
        break

rs = arcpy.RecordSet()
rs.load("http://sampleserver6.arcgisonline.com/arcgis/rest/services/ServiceRequest/FeatureServer/1/query?where=1%3D1&outFields=*&returnGeometry=true&f=json")
rsJSON = json.loads(rs.JSON)
with arcpy.da.SearchCursor(rs,"*") as cursor:
    for row in cursor:
        print(row)
        break‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The second layer in the ServiceRequest service is a table. The responses are:

(1118183, None, None, None, None, None, None, None, u'Unassigned', u'{F522A006-6BD4-432F-B55B-F52B6C48D8FC}', None, None, (None, None))
(310006, u'1', u'Obstructing stop sign', datetime.datetime(2018, 4, 5, 17, 15), 4, u'{1D161AE8-F54F-45B4-8EFC-4FCFEB42C76B}')‍‍‍‍

The breaks are in there to just get the first row.

Re: Using a AGOL HFS in Python directly?

DougBrowning — Fri, 20 Apr 2018 22:04:20 GMT

You code with your URLs works just fine for me.

If I use only your code and paste in my URL for a FC it works (with a added manually token).

If I change only the layer number from a FC layer to a table I get the Cannot load to table error again.

If I take the table URL that fails in python and paste it into the browser it works just fine.

If I take the result from the browser, add it as text to Python, then try a load from text I get the Error again!

I asked and I can maybe share it with you if you give the name to share with.

Weird.

Thanks