Hello,
I'm working on automating a process by creating a script that basically takes rows from a specified CSV file and adds them as features to an existing Hosted Feature Layer on ArcGIS Online.
I'd gotten this working previously, but as of the last few days, it's stopped working. The code runs, but nothing happens -- no new rows, no error message, nothing. So, I started checking components to see where the breakdown was.
My first check was the target layer pathway, "https://services.arcgis.com/[mapID]/arcgis/rest/services/[mapname]/FeatureServer/0", which, when copied and pasted into a browser, now says "Token Required". This is the same message I receive when I navigate to the page in browser, via Content > Feature Layer (Hosted) > Layers > URL (in the lower right) > View. If I click View directly (as an overlay) it works fine, but if I open it as a new tab, it presents the Token Required message. I'm familiar with this pathway for finding the accurate REST link despite that, but my understanding is that tokens are temporary, meaning hardcoding in one that was harvested from that approach would not work for long, undermining the point of automation. This does tell me that something is happening that is preventing direct connection to the feature layer, which is causing the script failure.
This suggested to me that perhaps something went wrong in the log-in or session ID phase of the code, so I tested some more, and I think something might be going on with the session.
We're a SAML (Azure AD) environment, but as per the documentation OAuth2.0 requires user interaction in its client_id approach (again defeating the purpose of automation), so I instead use my ArcGIS Pro portal connection to connect to my Enterprise ArcGIS Online environment, using
gis = GIS('pro')
which I then confirmed success of, using
print(gis.users.me)
print("Logged in as " + str(gis.properties.user.username))
both of which return the expected username value. This indicates (as far as I can tell) that the log-in was successful and the Pro portal pathway is working as expected. Furthermore, I can confirm session information, as
sessionInfo = gis._con._session
print(sessionInfo)
returns "<ArcGIS Session 0.1.0>" with no errors.
However, when I try to get a token based on this session, it fails.
token = gis.session.auth.token
returns an error as follows:
Traceback (most recent call last):
File "C:\Users\[username]\Downloads\1.py", line 13, in <module>
token = gis.session.auth.token
AttributeError: 'GIS' object has no attribute 'session'
I have not been able to find any information online about anyone else who has experienced this specific error. The confirmations of my username and session number suggest that there is indeed a successful session established by the log-in command, but... perhaps it's one that the script cannot fully access?
I've tried logging out of and back into ArcGIS Pro (Portal), logging out of and back into ArcGIS Online, clearing ArcGIS Pro's cache, and restarting my machine, to no avail.
What am I missing here? How can I get my Pro-portal login method to consistently generate a usable session so I can push data to my Hosted Layer?
Solved! Go to Solution.
Hey @ChrisThayer
Hmm that is strange, a built-in account is typically a specially created account, shown here:
This may be something that is turned off via settings, but creating an account like this avoids many of the domain login issues that seemingly plague the GIS() login methods that haven't been resolved or are poorly clarified in documentation.
I login using this method on a disconnected Visual Studio Code instance and can login and grab my tokens without issue. Along with this, you could also create a function that gets a token using something like this:
def get_bearer_token():
token_values = {
"username": "username",
"password": "password",
"client": "requestip",
"f": "json"
}
url = "https://company.com/arcgis/tokens/generateToken"
try:
response = requests.post(url, data=token_values)
if response.status_code != 200:
print(f"Request token failed: {response.status_code}")
return None
response_data = response.json()
return response_data.get("token")
except Exception as e:
print(f"An error occurred: {e}")
return None
token = get_bearer_token()
This would generate a token from the enterprise environment you're connected to, but a built-in account is almost necessary due to the previous reasons.
For those coming back to this in the future, I kept testing things in my code and consulted with a few folks, and figured out the core error (why things weren't updating) was actually coming from a different step in the process.
This does suggest, however, that whatever token or similar system the SAML path is using behind the scenes is probably not queryable in our code, so plan accordingly.
One thing that helped me determine that the problem wasn't with my connection to AGOL or my hosted layer was calling for a row count of the hosted layer -- because if it can call a row count successfully, then it is connected and the break was elsewhere. To do so,
import arcgis
import arcpy
from arcgis.gis import GIS
# Log in and confirm
gis = GIS('pro') #log into GIS via ArcGIS Pro portal connection
print("Logged in as " + str(gis.properties.user.username))
# Specify targeted hosted layer
update_definition_url = "https://services.arcgis.com/[mapID]/arcgis/rest/services/[mapName]/FeatureServer/0" # Plug in the secured URL
# Note that layer ID can be a bit tricky -- if you're getting 400 errors, you probably have the wrong layer ID
# To check, on AGOL go to the map Overview, under Layers [center right] click the *layer* name, then the little View button over the URL. At the top will say Layer: [name] (ID:#) -- that # is your layer number for this URL
# Get row count
arcpy.MakeFeatureLayer_management(update_definition_url, "myfeatures")
arcpy.SelectLayerByAttribute_management("myfeatures", "NEW_SELECTION")
resultTable1 = arcpy.GetCount_management("myfeatures")
print(resultTable1.getOutput(0)) #If it's pulling the right number of features, it definitely is connecting
I also wanted to include, for folks specifically struggling with adding new features and getting no error back but no new features either, there's a handy piece of code that didn't appear in most of the guides/Q&As I read, that lets you learn more about what happened to your attempt. It is
print(resultAppend['addResults']) #where resultAppend = layer_to_append.edit_features(adds=rows_to_append), layer_to_append being the existing hosted layer and rows_to_append the new data to add to it
This provided me the clue that there was a parsing problem with my underlying data that was causing the process to fail at that row and, crucially, the operation to be rolled back (thus no new rows appearing, even though some were successful).
I hope this helps folks in the future. I appreciate how much past questions on this board have been a huge help to me and wanted to pay it forward. Thank you, regular contributors!
Hey @ChrisThayer
Would this here work to get your token:
token = gis._con.token
print(token)
I'm curious if you would be able to log into the arcgis method with GIS('username', 'password', 'arcgis online link') and go through there instead of GIS('pro') with a built-in account instead of SAML?
Cody
I tried the provided code; it runs, but returns the message "None." as the token value.
During testing, I experimented with logging in directly with username and password, but from what I can tell, we don't have built-in accounts, only SAML. In case I was getting the code wrong, I first tried directly logging into ArcGIS Online (via browser) using the username and password boxes (instead of the enterprise button). I even tested it using a couple different variants of my username. In all cases, it rejected them as unrecognized -- which makes me think there's probably no account directly in the ArcGIS system, only the pass-through account via SAML.
Hey @ChrisThayer
Hmm that is strange, a built-in account is typically a specially created account, shown here:
This may be something that is turned off via settings, but creating an account like this avoids many of the domain login issues that seemingly plague the GIS() login methods that haven't been resolved or are poorly clarified in documentation.
I login using this method on a disconnected Visual Studio Code instance and can login and grab my tokens without issue. Along with this, you could also create a function that gets a token using something like this:
def get_bearer_token():
token_values = {
"username": "username",
"password": "password",
"client": "requestip",
"f": "json"
}
url = "https://company.com/arcgis/tokens/generateToken"
try:
response = requests.post(url, data=token_values)
if response.status_code != 200:
print(f"Request token failed: {response.status_code}")
return None
response_data = response.json()
return response_data.get("token")
except Exception as e:
print(f"An error occurred: {e}")
return None
token = get_bearer_token()
This would generate a token from the enterprise environment you're connected to, but a built-in account is almost necessary due to the previous reasons.
Thanks @CodyPatterson ! It's very reassuring to know that at least there are known issues with this method and it's not just me missing something obvious in the documentation.
I'm not an admin on our instance, so I don't think I can create a new account in it, to be able to make a built-in account. I gave the provided code a go with my best guess at the relevant username and password (for my organizational account), and it returned "An error occurred: name 'requests' is not defined".
For clarity, at the top of my code I'm calling in the following (in case there's something else that needs to be imported):
import json
from arcgis.gis import GIS
from arcgis.features import Feature
from arcgis.features import FeatureLayer
from arcgis.geometry import Point
Hey @ChrisThayer
To get requests in there you'd just need to add:
import requests
This should get that function going, but without the proper username and password it may still continue to error, if you have a GIS Admin that creates the accounts, it may be worth to request a user to be created for the purposes of programming like this, especially if you are a developer.
Cody
I added it. Now I'm getting a response of "Request token failed: 404". Does that mean File Not Found in this context too? And/or, does that line up with the suspected username issue?
I'll contact the GIS admin staff at my organization to see if it's possible to get such a thing set up (though it'll probably take a while even if it is, so there probably won't be an update about it here). It's just so odd to me that my script worked perfectly with my enterprise account for a couple of days and then just stopped, out of nowhere.
For those coming back to this in the future, I kept testing things in my code and consulted with a few folks, and figured out the core error (why things weren't updating) was actually coming from a different step in the process.
This does suggest, however, that whatever token or similar system the SAML path is using behind the scenes is probably not queryable in our code, so plan accordingly.
One thing that helped me determine that the problem wasn't with my connection to AGOL or my hosted layer was calling for a row count of the hosted layer -- because if it can call a row count successfully, then it is connected and the break was elsewhere. To do so,
import arcgis
import arcpy
from arcgis.gis import GIS
# Log in and confirm
gis = GIS('pro') #log into GIS via ArcGIS Pro portal connection
print("Logged in as " + str(gis.properties.user.username))
# Specify targeted hosted layer
update_definition_url = "https://services.arcgis.com/[mapID]/arcgis/rest/services/[mapName]/FeatureServer/0" # Plug in the secured URL
# Note that layer ID can be a bit tricky -- if you're getting 400 errors, you probably have the wrong layer ID
# To check, on AGOL go to the map Overview, under Layers [center right] click the *layer* name, then the little View button over the URL. At the top will say Layer: [name] (ID:#) -- that # is your layer number for this URL
# Get row count
arcpy.MakeFeatureLayer_management(update_definition_url, "myfeatures")
arcpy.SelectLayerByAttribute_management("myfeatures", "NEW_SELECTION")
resultTable1 = arcpy.GetCount_management("myfeatures")
print(resultTable1.getOutput(0)) #If it's pulling the right number of features, it definitely is connecting
I also wanted to include, for folks specifically struggling with adding new features and getting no error back but no new features either, there's a handy piece of code that didn't appear in most of the guides/Q&As I read, that lets you learn more about what happened to your attempt. It is
print(resultAppend['addResults']) #where resultAppend = layer_to_append.edit_features(adds=rows_to_append), layer_to_append being the existing hosted layer and rows_to_append the new data to add to it
This provided me the clue that there was a parsing problem with my underlying data that was causing the process to fail at that row and, crucially, the operation to be rolled back (thus no new rows appearing, even though some were successful).
I hope this helps folks in the future. I appreciate how much past questions on this board have been a huge help to me and wanted to pay it forward. Thank you, regular contributors!