Upload local content (.xlsx, feature class) to /arcgis/home workspace for AGOL Notebook

[JesseHandler]

Hello,

I have a currently working script that I have been running in ArcGIS Pro to update data in a hosted feature layer with data from an excel table stored on an organization server. I am trying to transfer the script over to AGOL notebooks so that I can make use of the tasks function and schedule the script to run on a nightly basis to update the layer's data. However, I am having trouble figuring out how to use the local table stored on my organization's server in the script. I know I can manually upload my excel file to the /arcgis/home workspace, but cannot find sample code for how to upload my data files. Any code suggestions? Below is a portion of the working code in my ArcGIS Pro Notebook that I am trying to replace in an AGOL Notebook.

[curtvprice]

The AGOL platform can't see local files (ie your M drive).

The only way to make this work is to upload your excel data to a table in Google Sheets or as a csv to ArcGIS Online. The latter task could be done on the Windows side using a python script that uses the gis module. Here is a learn.arcgis.com  tutorial to update a feature service, try that, if you get it working you can use the same procedure to update your .csv.

Another possibility is copying your excel sheet to a web accessible folder so you can provide an https address to read the excel file, but I am unsure that would work.

[RaviNarayanan]

hi @JesseHandler 

Suggestions above from @curtvprice will work. Alternately Notebooks have a Data access API which can be used to upload files to the workspace directory. The team is looking into adding support for Data Acces API in ArcGIS API for Python in a future release. In the mean team time here is the workflow to upload a file to the workspace directory described via curl commands.  You can use this API info to code this in programming language of choice. 

Step 1: Generate a token for the user (replace <username> and <password> with appropriate values. don't use <>)

curl -d 'username=<username>&password=<password>&f=json&client=referer&referer=https' https://www.arcgis.com/sharing/rest/generateToken

This will return a response as below. You will need the token value for the requests below.

{"token":"6bkID9qoDHljg6zH3FX4ktUema_pFDme_hB-IK9-EtTugjamw-hntVaETno.","expires":1624653125658,"ssl":true}

Step 2: Obtain the data access API endpoint for your AGOL organization. Replace <token> and <username> with appropriate values.

curl -d 'token=<token>&f=json&storeType=notebookWorkspace' -H 'Referer: https' https://www.arcgis.com/sharing/rest/content/users/<username>/generateDirectAccessUrl

This will return a response as below specific to your AGOL org. You will use the url for uploading the file.

{"url":"https://notebooksdataaccess9.arcgis.com/yourorgid/arcgis/admin/dataaccess/azureblob/notebooksWorkspace"}

Step 3: HTTP PUT the file in your workspace. In the example below, I am uploading a file named 1c.png located in my local machines Downloads directory.  Please note the upload url returned from request above is suffixed with the desired folder path\file name. In this case below, the file will be uploaded to the root workspace  (/arcgis/home) directory. 

curl -T ~/Downloads/1c.png -H 'Referer: https' -H 'Content-Type: application/octet-stream' -H 'x-ms-blob-type: BlockBlob' -H 'x-ms-version: 2020-02-10' https://notebooksdataaccess9.arcgis.com/myorgid/arcgis/admin/dataaccess/azureblob/notebooksWorkspace/1c.png?token=<token>

[JesseHandler]

Hi @RaviNarayanan ,

Thank you so much for posting the above solution using curl commands. I have been able to get step 1 and 2 working, but I am having trouble with step 3. I have converted your code to python that looks like below (myorg in url has been replaced with the correct org number).

#HTTP PUT the file in your workspace
headers2 = {
    'Referer': 'https',
    'Content-Type': 'application/octet-stream',
    'x-ms-blob-type': 'BlockBlob',
    'x-ms-version': '2020-02-10',
}

params = {
    'token': x["token"]
}

#option 1:
files = {'file_name': open(r"C:\Users\jhandler\Desktop\ArcGISforPythonAPI_Test\Unit_Level_Detail.xlsx",'rb')}
#option 2:
files = {'file_name': open("C:/Users/jhandler/Desktop/ArcGISforPythonAPI_Test/Unit_Level_Detail.xlsx",'r').read()}

response3 = requests.put("https://notebooksdataaccess9.arcgis.com/myorgid/arcgis/admin/dataaccess/azureblob/notebooksWorkspace/Unit Level Detail.xlsx",
                         files=files,
                         headers=headers2, 
                         params=params)

 I am getting the following error message of File Not Found when I run it with Option #1 and Option #2 for the files parameter.

FileNotFoundError                         Traceback (most recent call last)
<ipython-input-15-6905e1421eb1> in <module>
     12 
     13 #files = {'file_name': open(r"C:\Users\jhandler\Desktop\ArcGISforPythonAPI_Test\Unit_Level_Detail.xlsx",'rb')}
---> 14 files = {'file_name': open("C:/Users/jhandler/Desktop/ArcGISforPythonAPI_Test/Unit_Level_Detail.xlsx",'r').read()}
     15 
     16 response3 = requests.put("https://notebooksdataaccess9.arcgis.com/myorgid/arcgis/admin/dataaccess/azureblob/notebooksWorkspace/Unit Level Detail.xlsx",

FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/jhandler/Desktop/ArcGISforPythonAPI_Test/Unit_Level_Detail.xlsx'

Any suggestions for how to reference this local file? I am running the code in AGOL notebooks.

[RaviNarayanan]

Hi @JesseHandler 

This will not work from a notebook in AGOL as it will not be able to access local content..can you try running this python code locally?