Hi,
I would like to automate the update of a hosted feature layer by uploading new source data in the form of a zipped file geodatabase. Currently, I'm uploading a modified statewide cadastre dataset weekly using the ArcGIS online user interface, but I would like to integrate that process into a python script.
Has anyone had a similar need and already developed a solution?
Cheers,
Bernhard
You can overwrite the entire FS using this (https://blogs.esri.com/esri/arcgis/2017/03/14/updating-your-hosted-feature-services-with-arcgis-pro-...) which is more "friendly" to error handling, or this (https://developers.arcgis.com/python/sample-notebooks/updating-features-in-a-feature-layer/) which would involve a little more complexity, and does not handle data deltas.
import sys, string, os, arcpy, calendar, datetime, traceback,smtplib
import arcpy
from arcpy import env
from arcgis.gis import GIS
from subprocess import call
# Set the name of the feature service. This will control many variable in this script
NAME = "HFS_NAME"
# Define log setting
try:
d = datetime.datetime.now()
log = open("C:\\PYTHON_LOGS\LOG."+NAME+".txt","a")
log.write("----------------------------" + "\n")
log.write("----------------------------" + "\n")
log.write("Log: " + str(d) + "\n")
log.write("\n")
# Start process...
starttime = datetime.datetime.now()
log.write("Begin process:\n")
log.write(" Process started at " + str(starttime) + "\n")
log.write("\n")
### Start setting variables
# Mail Server Settings
SERVER = "mail_server"
PORT = "25"
FROM = "sasquatch"
MAILDOMAIN = '@big.foot'
# Data Steward getting the email. Needs to be their email address...without @big_foot at the end
userList=["is_real"]
# get a list of usernames from the list of named tuples returned from ListUsers
userNames = [u for u in userList]
# take the userNames list and make email addresses by appending the appropriate suffix.
emailList = [name + MAILDOMAIN for name in userNames]
TO = emailList
# Grab date for the email
DATE = d
#Delete any left over SD files from failed previous run
sdfilename = NAME+".sd"
sddraftfilename = NAME+".sddraft"
if os.path.exists(sdfilename): os.remove(sdfilename)
if os.path.exists(sddraftfilename):os.remove(sddraftfilename)
# Set the path to the ArcGIS Pro project
prjPath = r"C:\PRODUCTION\HFS_NAME.aprx"
# Update the following variables to match:
# Feature service/SD name in arcgis.com, user/password of the owner account
sd_fs_name = NAME
portal = "http://www.arcgis.com" # Can also reference a local portal
user = "users"
password = "pass"
# Set sharing options
shrOrg = True
shrEveryone = True
shrGroups = "some group id "
### End setting variables
# Local paths to create temporary content
relPath = sys.path[0]
sddraft = os.path.join(relPath, NAME+".sddraft")
sd = os.path.join(relPath, NAME+".sd")
# Create a new SDDraft and stage to SD
print("Creating SD file")
arcpy.env.overwriteOutput = True
prj = arcpy.mp.ArcGISProject(prjPath)
mp = prj.listMaps()[0]
arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, sd_fs_name, 'MY_HOSTED_SERVICES', 'FEATURE_ACCESS','', True, True)
arcpy.StageService_server(sddraft, sd)
print("Connecting to {}".format(portal))
gis = GIS(portal, user, password)
# Find the SD, update it, publish /w overwrite and set sharing and metadata
print("Search for original SD on portal...")
#sdItem = gis.content.search("title:{} AND owner:{}".format(sd_fs_name, user), item_type="Service Definition")[0]
#sdItem = gis.content.search(query="title:"+ sd_fs_name + " AND owner: " + user)[0]
#sdItem = gis.content.search("id:som sd id")
sdItem = gis.content.search("title:HFS_NAME AND owner:user AND id:some sd id", item_type="Service Definition")[0]
print("Found SD: {}, ID: {} \n Uploading and overwriting...".format(sdItem.title, sdItem.id))
sdItem.update(data=sd)
print("Overwriting existing feature service...")
fs = sdItem.publish(overwrite=True)
if shrOrg or shrEveryone or shrGroups:
print("Setting sharing options...")
fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)
print("Finished updating: {} - ID: {}".format(fs.title, fs.id))
# Clean up SD files
os.remove(sd)
os.remove(sddraft)
# Write nothing to log if success.
endtime = datetime.datetime.now()
log.write(" Completed successfully in "
+ str(endtime - starttime) + "\n")
log.write("\n")
log.close()
print('done')
# Something with wrong
except:
# Get the traceback object
tb = sys.exc_info()[2]
tbinfo = traceback.format_tb(tb)[0]
# Concatenate information together concerning
# the error into a message string
pymsg = "PYTHON ERRORS:\nTraceback info:\n" + tbinfo + "\nError Info:\n" + str(sys.exc_info()[1])
msgs = "ArcPy ERRORS:\n" + arcpy.GetMessages(2) + "\n"
# Return python error messages for use in
# script tool or Python Window
arcpy.AddError(pymsg)
arcpy.AddError(msgs)
# Print Python error messages for use in
# Python / Python Window
log.write("" + pymsg + "\n")
log.write("" + msgs + "")
log.close()
# Define email message if something went wrong
SUBJECT = "Notification of Un-Successful AGOL Update of "+NAME
MSG = "Did Not Update: {} - ID: {} at "+ str(DATE)+ "; " +pymsg + "; " + msgs
print (MSG)
print (emailList)
# Send an email notifying steward of successful archive
#MESSAGE = "\ From: %s To: %s Subject: %s %s" % (FROM, ", ".join(TO), SUBJECT, MSG)
MESSAGE = "Subject: %s\n\n%s" % (SUBJECT, MSG)
try:
try:
print("Connecting to Server...")
server = smtplib.SMTP(SERVER,PORT)
try:
print("Login...")
try:
print("Sending mail...")
server.sendmail(FROM, TO, MESSAGE)
except Exception as e:
print("Send Error Mail\n" + e.message)
except Exception as e:
print("Error Authentication Server: check the credentials \n" + e.message)
except Exception as e:
print("Error Connecting to Server : check the URL of the server and communications port ( 25 and ' the default ) \n" + e.message)
print("Quit.")
server.quit()
except Exception as e:
print (e.message)
You can also try this sample (older, written in python 2.7) and remove the editor tracking bits if this isn't needed:
http://cloudygis.maps.arcgis.com/home/item.html?id=2d7027b148e24005916e55ab2cb74b50
Also, try taking a look at the ArcGIS Python API
Add Item (upload FGDB): arcgis.gis module — arcgis 1.3.0 documentation
Publish Item including overwrite: arcgis.gis module — arcgis 1.3.0 documentation
-Kelly
Thomas,
For clarification, will this code example replace the pre-existing data or append new data to it?
Also, what version of Python was your example written in?
I am looking to append data (new event data) to a pre-existing database in ArcGIS Online, similar to what you can do manually. However, I want to automate the process.
Thanks,
Can we run this code without having to store our username and password in plain text?
Sure,
You can modify how you add the username and password using any standard python method. In order to run the script above, you will need to somehow generate a token (requires username and password) and then append it to all of the required requests.