topic Re: Logged into AGOL via Python API but can't access my items in ArcGIS API for Python Questions

Logged into AGOL via Python API but can't access my items

Anonymous User — Sun, 12 Dec 2021 06:41:42 GMT

When connecting to my organizational AGOL account via the Python API, I have found lately that while the login appears to succeed, I do not have the same level of access to my items that I have when accessing them through the AGOL browser interface or ArcGIS Pro.

Setup

OS: Windows 10
Python Version: 3.5 on Anaconda.
NOTE: This is not the an environment from the Anaconda installation that comes with ArcGIS Pro. This is an installation I had prior to installing ArcGIS Pro. (requirements.txt attached)
ArcGIS Python API Version: 1.4.2

First, I import my packages:

from arcgis.gis import *
from IPython.display import display‍‍‍‍‍‍‍

My organizational account must be authenticated via OAuth 2.0. I have followed the instructions here for authenticating with OAuth 2.0. I have created an application called Python and used the following code to login.

gis = GIS("https://{{ MY_ORG }}.maps.arcgis.com", client_id='{{MY_APP_ID}}')‍
print("Successfully logged in as: " + gis.properties.user.username)‍‍

This appears to work with no problem as there are no errors returned and the following statement is printed:

Successfully logged in as: {{ MY USERNAME }}‍‍

However, problems arise when I try to access any of my items. Note, by "my items" I mean those for which I am the item owner. For the sake of example, let's say I try to access an item by its item number:

item = gis.content.get('{{SOME ITEM ID}}')
display(item)‍‍‍‍

This throws the following error

You do not have permissions to access this resource or perform this operation.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-faf871f03a5b> in <module>()
----> 1 gis.content.get('{{ SOME ITEM ID }}')

~\Miniconda3\envs\arcpython\lib\site-packages\arcgis\gis\__init__.py in get(self, itemid)
   2653                 return None
   2654             else:
-> 2655                 raise re
   2656 
   2657         if item is not None:

~\Miniconda3\envs\arcpython\lib\site-packages\arcgis\gis\__init__.py in get(self, itemid)
   2648         """
   2649         try:
-> 2650             item = self._portal.get_item(itemid)
   2651         except RuntimeError as re:
   2652             if re.args[0].__contains__("Item does not exist or is inaccessible"):

~\Miniconda3\envs\arcpython\lib\site-packages\arcgis\_impl\portalpy.py in get_item(self, itemid)
   1206             ================  ========================================================
   1207         """
-> 1208         return self.con.post('content/items/' + itemid, self._postdata())
   1209 
   1210     def get_item_data(self, itemid, try_json=True):

~\Miniconda3\envs\arcpython\lib\site-packages\arcgis\_impl\connection.py in post(self, path, postdata, files, ssl, compress, is_retry, use_ordered_dict, add_token, verify_cert, token, try_json, out_folder, file_name, force_bytes, add_headers)
   1154                                          verify_cert=verify_cert, is_retry=True)
   1155 
-> 1156                 self._handle_json_error(resp_json['error'], errorcode)
   1157                 return None
   1158 

~\Miniconda3\envs\arcpython\lib\site-packages\arcgis\_impl\connection.py in _handle_json_error(self, error, errorcode)
   1175 
   1176         errormessage = errormessage + "\n(Error Code: " + str(errorcode) +")"
-> 1177         raise RuntimeError(errormessage)
   1178 
   1179 class _StrictURLopener(request.FancyURLopener):

RuntimeError: You do not have permissions to access this resource or perform this operation.
(Error Code: 403)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I have tested this process under a few different scenarios and not had these permission errors arise:

On the same machine as this example, if I use the arcgispro-py3 environment with the Anaconda installation that comes with ArcGIS Pro, I am able to authenticate and access my items.
Again, on the same machine as this example, if I have ArcGIS Pro running, I can use Pro to authenticate. However, I must also be using the arcgispro-py3 environment with the Anaconda installation that comes with ArcGIS Pro.
Using an entirely different machine running Ubuntu, I created the same environment OAuth 2.0 process as above and was successful in accessing my content.

Has anyone encountered this issue before? Any thoughts on how to troubleshoot?

Re: Logged into AGOL via Python API but can't access my items

Zach_Robinson — Fri, 27 Jul 2018 21:14:23 GMT

Hi Ryan, I was having the same frustrations as you and I ended up calling ESRI for assistance. Basically, it doesn't matter what your organization privileges are or even if you own the data. The item must be shared publicly in order to be accessible via gis.content.get

Even the analyst on the phone agreed with me that seems to restrict the ability to use the API as an organizational administrator but that is how the API is currently written.

Re: Logged into AGOL via Python API but can't access my items

Anonymous User — Mon, 30 Jul 2018 13:48:16 GMT

Thanks for sharing your experience. I have managed to use gis.content.get to access private items using the username and password of a public account associated with our organization. Not really ideal because almost nobody in our organization authenticates this way. Especially since our org is going all-in on AGOL, it's frustrating to find out that ownership doesn't really mean ownership.

Re: Logged into AGOL via Python API but can't access my items

MichaelKelly — Sun, 12 Dec 2021 06:41:45 GMT

The following code displays private items for me:

from IPython.display import display
from arcgis.gis import GIS

ItemId = 'bc9f80dd6ea341f9b420i1998c07ec05'
PortalUserName = 'myCaseSensitiveUsername'
PortalPassword = 'myPassword'
PortalUrl = 'https://www.arcgis.com'
PortalCertVerification = True

gis = GIS(PortalUrl, PortalUserName, PortalPassword, verify_cert=PortalCertVerification)
itemObject = gis.content.get(ItemId)
display(itemObject)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

As you mentioned it is absolutely possible to access, display and modify private content via the API, and you have it working in various environments.

However this particular case is a little different because of the kind of authentication you are using. When you login using OAuth this way by registering an application, you don't gain access to all content owned by the user - there are various built in limitations, as documented here (also listed below):

Tokens obtained by applications can only read public content and services. Although you cannot use an App login with private content, if your goal is to distribute or sell an app to organizations without ArcGIS Online (no named users), you may control access to your content by using your own login mechanism (I.e. Identity) to the app.
Tokens obtained by applications may read premium content and services hosted by Esri and consume credits on behalf of the application organization.
Applications cannot create, update, share, modify, or delete items (layers, files, services, maps) in ArcGIS Online or ArcGIS Enterprise.
Applications built using app login cannot be listed in the ArcGIS Marketplace.

This is most likely the cause of the permissions message you are getting, and is by design. In order to access private content, you will have to work with one of the other authentication schemes available with the API.

Re: Logged into AGOL via Python API but can't access my items

Anonymous User — Tue, 07 Aug 2018 13:54:28 GMT

Thank you for your response, Michael. We use ADFS to authenticate, so the example you've include unfortunately doesn't apply to my organization. I have tried adapting the the Active Directory instructions on the link you included, but have been unable to successfully authenticate. I've had trouble determining what the webadapter name and domain are. Regardless OAuth through an app is the only authentication scheme I've gotten to work.

Is there an explanation for why this restriction on tokens obtained by applications exists? I'm just a little confused as to why this sort of authentication is sufficient for me to access premium content available through my organizations ELA, but not enough for me to access items that are mine.

Re: Logged into AGOL via Python API but can't access my items

Anonymous User — Sun, 12 Dec 2021 06:41:48 GMT

Following up on my attempt to authenticate using the Active Directory.

Here's what is listed on the website about authenticating using the Python API:

from arcgis.gis import GIS

gis = GIS("https://portalname.domain.com/webadapter_name", "AVWORLD\\Publisher", "password")‍‍‍

I have to admit I don't have the language to speak to the difference between Portal and...well I'm not sure what else...but what I can say is all our data, maps, and apps are on ArcGIS Online as a part of our Organization. As for domain, I have used the one found under my Windows system information:

So based on the template above, I thought I'd be able to authenticate with the following:

gis = GIS("https://ral.maps.arcgis.com/arcgis", "ci.raleigh.nc.us\\{USERNAME}", "{PASSWORD}")‍

However, I just get a 404 error.

I've spoken with our IT/GIS department and they seemed to thing I was using the correct web adaptor and domain names, but still no dice. Any recommendations?

Re: Logged into AGOL via Python API but can't access my items

MichaelKelly — Sun, 12 Dec 2021 06:41:51 GMT

Hi Ryan,

I think you're nearly there. The Portal URL you are providing does not exist (hence the 404), you need something like this (note 'home' vs 'arcgis'):

from arcgis.gis import GIS

PortalUsername = 'ci.raleigh.nc.us\\yourUsername'
PortalPassword = 'yourPassword'
PortalUrl = 'https://ral.maps.arcgis.com/home'

gis = GIS(PortalUrl, PortalUsername, PortalPassword)

print("Logged in as: " + gis.properties.user.username)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Any joy with this? An alternative would be to use an ArcGIS Online Named User which doesn't use ADFS.

The aim of app logins is to allow the use of premium content in ArcGIS Online without having to sign in. Essentially it is an authentication mechanism which allows the use of premium services publically (credits are consumed by the app login user where appropriate). It is not intended to grant access to content owned by the user in question. It is typically used in combination with various API's/SDK's.

Mikie

Re: Logged into AGOL via Python API but can't access my items

MichaelKelly — Thu, 09 Aug 2018 15:27:31 GMT

I would also suggest you try connecting via ArcGIS Pro:

gis = GIS("pro")‍

This will use the active user in ArcGIS Pro.

Re: Logged into AGOL via Python API but can't access my items

AllWeatherHeather — Thu, 27 Dec 2018 15:47:53 GMT

Ryan - what did you end up doing to get this to work for you? I'm in the same boat with ADFS as the enterprise standard.

Re: Logged into AGOL via Python API but can't access my items

Anonymous User — Wed, 02 Jan 2019 14:01:58 GMT

Heather-

I wish I had better news, but unfortunately I was never able to successfully authenticate using the Python API via ADFS.

Re: Logged into AGOL via Python API but can't access my items

NareshAligeti1 — Mon, 19 Aug 2019 21:35:52 GMT

Ryan,

Any luck authenticating AGOL with ADFS using Python API? Did you end up having plan B?

Re: Logged into AGOL via Python API but can't access my items

davedoesgis — Fri, 24 Jul 2020 23:49:28 GMT

Michael Kelly‌

I am getting this error just accessing the shared_with property of an Item (a hosted feature layer, in this case). It is very strange, because the same code does not have this problem with a different hosted feature layer. This is a property of the item and should not be reaching out to AGOL to access it, so my gut is telling me that this is NOT some AGOL permission error, despite the error output text. (At least in my case, not sure about the original post.)

When I stop the code in debug mode, my debugger says it is unable to get repr (i.e.: string representation) for the shared_with property, it's groups list and the 2nd member of that list. The org and everyone bool elements that look right. The item is shared with 2 groups (the 1st and 3rd with the names crossed out), but for some reason, there is a 3rd group in the list.

I'm not sure if my problem is the same as yours, but this feels like an Esri bug, not some actual permissions error. I got the same error just trying to print the item.shared_with value.

See attached file for the errors I was able to produce when I stopped my code in debug mode using Eclipse PyDev as my IDE. I poked at the object and was able to produce the error trying to access each of the yellow highlighted elements in my screenshot. Here is the short version:

>>> item
<Item title:"<_redacted_>" type:Feature Layer Collection owner:<_redacted_>>
>>> item.shared_with
Traceback (most recent call last):
File "C:\eclipse\plugins\org.python.pydev.core_7.6.0.202006041357\pysrc\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
exec(exp, global_vars, local_vars)
File "<console>", line 1, in <module>
File "C:\Users\username\AppData\Local\ESRI\conda\envs\dev\lib\site-packages\arcgis\gis\__init__.py", line 5562, in __repr__
return '<%s title:"%s" owner:%s>' % (type(self).__name__, self.title, self.owner)
File "C:\Users\username\AppData\Local\ESRI\conda\envs\dev\lib\site-packages\arcgis\gis\__init__.py", line 5540, in __getattr__
self._hydrate()
File "C:\Users\username\AppData\Local\ESRI\conda\envs\dev\lib\site-packages\arcgis\gis\__init__.py", line 5533, in _hydrate
groupdict = self._portal.get_group(self.groupid)
File "C:\Users\username\AppData\Local\ESRI\conda\envs\dev\lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 906, in get_group
return self.con.post('community/groups/' + group_id, self._postdata())
File "C:\Users\username\AppData\Local\ESRI\conda\envs\dev\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 694, in post
force_bytes=kwargs.pop('force_bytes', False))
File "C:\Users\username\AppData\Local\ESRI\conda\envs\dev\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 493, in _handle_response
self._handle_json_error(data['error'], errorcode)
File "C:\Users\username\AppData\Local\ESRI\conda\envs\dev\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 511, in _handle_json_error
raise Exception(errormessage)
Exception: You do not have permissions to access this resource or perform this operation.
(Error Code: 403)

Re: Logged into AGOL via Python API but can't access my items

Anonymous User — Mon, 07 Dec 2020 17:27:13 GMT

More than 2 years after asking this question and forgetting about it, I happened upon this thread while troubleshooting a similar inability to access non-public items that are either mine or accessible to me via sharing. I'm better at Python now so I was able to hack together a workaround that may be useful for others. Basically, it involves circumventing the ArcGIS Python API and instead using requests to return a JSON response with your data. From there, you can use pandas or, in my case, geopandas, to get the data in a more table-like format.

import requests import json from arcgis.gis import GIS gis = GIS("https://myorg.maps.arcgis.com", client_id='abc123', verify_cert = True) layer_url = 'https://services.arcgis.com/myorg/arcgis/rest/services/my_layer/FeatureServer/0/query' layer_params = { "where": "1=1", "f": "geojson", "outSR": 4326, "outFields": "*", "token": gis._con.token } layer_r = requests.get(layer_url, layer_params) layer_json = json.dumps(layer_r.json())

I still don't understand why authenticating with OAuth2 has any bearing on accessing private items through the Python API, but if you're just trying to get at the data in a non-public layer, this may work for you.