Automating the update of a hosted feature layer with new data, e.g. from a zipped file geodatabase, using a python script.

10071
5
01-01-2018 11:45 PM
BernhardKlingseisen2
New Contributor

Hi,

I would like to automate the update of a hosted feature layer by uploading new source data in the form of a zipped file geodatabase. Currently, I'm uploading a modified statewide cadastre dataset weekly using the ArcGIS online user interface, but I would like to integrate that process into a python script.

Has anyone had a similar need and already developed a solution?

Cheers,

Bernhard     

0 Kudos
5 Replies
ThomasColson
MVP Frequent Contributor

You can overwrite the entire FS using this (https://blogs.esri.com/esri/arcgis/2017/03/14/updating-your-hosted-feature-services-with-arcgis-pro-...) which is more "friendly" to error handling, or this (https://developers.arcgis.com/python/sample-notebooks/updating-features-in-a-feature-layer/) which would involve a little more complexity, and does not handle data deltas.

import sys, string, os, arcpy, calendar, datetime, traceback,smtplib
import arcpy
from arcpy import env
from arcgis.gis import GIS
from subprocess import call
# Set the name of the feature service. This will control many variable in this script
NAME = "HFS_NAME"
# Define log setting
try:
    d = datetime.datetime.now()
    log = open("C:\\PYTHON_LOGS\LOG."+NAME+".txt","a")
    log.write("----------------------------" + "\n")
    log.write("----------------------------" + "\n")
    log.write("Log: " + str(d) + "\n")
    log.write("\n")
# Start process...
    starttime = datetime.datetime.now()
    log.write("Begin process:\n")
    log.write("     Process started at " + str(starttime) + "\n")
    log.write("\n")

    
    ### Start setting variables
    # Mail Server Settings
    SERVER = "mail_server"
    PORT = "25"
    FROM = "sasquatch"
    MAILDOMAIN = '@big.foot'
    # Data Steward getting the email. Needs to be their email address...without @big_foot at the end
    userList=["is_real"]
    # get a list of usernames from the list of named tuples returned from ListUsers
    userNames = [u for u in userList]
    # take the userNames list and make email addresses by appending the appropriate suffix.
    emailList = [name +  MAILDOMAIN for name in userNames]
    TO = emailList
    # Grab date for the email
    DATE = d
    #Delete any left over SD files from failed previous run
    sdfilename = NAME+".sd"
    sddraftfilename = NAME+".sddraft"
    if os.path.exists(sdfilename): os.remove(sdfilename)
    if os.path.exists(sddraftfilename):os.remove(sddraftfilename)
    
    # Set the path to the ArcGIS Pro project
    prjPath = r"C:\PRODUCTION\HFS_NAME.aprx"
     
    # Update the following variables to match:
    #  Feature service/SD name in arcgis.com, user/password of the owner account
    sd_fs_name = NAME
    portal = "http://www.arcgis.com" # Can also reference a local portal
    user = "users"
    password = "pass"
 
    # Set sharing options
    shrOrg = True
    shrEveryone = True
    shrGroups = "some group id "
     
    ### End setting variables
     
    # Local paths to create temporary content
    relPath = sys.path[0]
    sddraft = os.path.join(relPath, NAME+".sddraft")
    sd = os.path.join(relPath, NAME+".sd")
     
    # Create a new SDDraft and stage to SD
    print("Creating SD file")
    arcpy.env.overwriteOutput = True
    prj = arcpy.mp.ArcGISProject(prjPath)
    mp = prj.listMaps()[0]
    arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, sd_fs_name, 'MY_HOSTED_SERVICES', 'FEATURE_ACCESS','', True, True)
    arcpy.StageService_server(sddraft, sd)
     
    print("Connecting to {}".format(portal))
    gis = GIS(portal, user, password)
 
    # Find the SD, update it, publish /w overwrite and set sharing and metadata
    print("Search for original SD on portal...")
    #sdItem = gis.content.search("title:{} AND owner:{}".format(sd_fs_name, user), item_type="Service Definition")[0]
    #sdItem = gis.content.search(query="title:"+ sd_fs_name + " AND owner: " + user)[0]
    #sdItem = gis.content.search("id:som sd id")
    sdItem = gis.content.search("title:HFS_NAME AND owner:user AND id:some sd id", item_type="Service Definition")[0]
    print("Found SD: {}, ID: {} \n Uploading and overwriting...".format(sdItem.title, sdItem.id))
    sdItem.update(data=sd)
    print("Overwriting existing feature service...")
    fs = sdItem.publish(overwrite=True)
     
    if shrOrg or shrEveryone or shrGroups:
      print("Setting sharing options...")
      fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)
     
    print("Finished updating: {} - ID: {}".format(fs.title, fs.id))

    # Clean up SD files
    os.remove(sd)
    os.remove(sddraft)
    # Write nothing to log if success.
    endtime = datetime.datetime.now()
    log.write("     Completed successfully in " 
           + str(endtime - starttime) + "\n")
    log.write("\n")
    log.close()
    print('done')
# Something with wrong
except:
    
 # Get the traceback object
    tb = sys.exc_info()[2]
    tbinfo = traceback.format_tb(tb)[0]
 # Concatenate information together concerning 
 # the error into a message string
    pymsg = "PYTHON ERRORS:\nTraceback info:\n" + tbinfo + "\nError Info:\n" + str(sys.exc_info()[1])
    msgs = "ArcPy ERRORS:\n" + arcpy.GetMessages(2) + "\n"
# Return python error messages for use in 
# script tool or Python Window
    arcpy.AddError(pymsg)
    arcpy.AddError(msgs)
# Print Python error messages for use in 
# Python / Python Window
    log.write("" + pymsg + "\n")
    log.write("" + msgs + "")
    log.close()
    # Define email message if something went wrong
    SUBJECT = "Notification of Un-Successful AGOL Update of "+NAME 
    MSG = "Did Not Update: {} - ID: {} at "+ str(DATE)+ "; " +pymsg + "; " + msgs
    print (MSG)
    print (emailList)
 
    # Send an email notifying steward of successful archive
    #MESSAGE = "\ From: %s To: %s Subject: %s %s" % (FROM, ", ".join(TO), SUBJECT, MSG)
    MESSAGE = "Subject: %s\n\n%s" % (SUBJECT, MSG)
    try:
            try:
                print("Connecting to Server...")
                server = smtplib.SMTP(SERVER,PORT)
                try:
                    print("Login...")
                    try:
                        print("Sending mail...")
                        server.sendmail(FROM, TO, MESSAGE)
                    except Exception as e:
                        print("Send Error Mail\n" + e.message)
                except Exception as e:
                    print("Error Authentication Server: check the credentials \n" + e.message)
            except Exception as e:
                print("Error Connecting to Server : check the URL of the server and communications port ( 25 and ' the default ) \n" + e.message)
     
            print("Quit.")
            server.quit()
     
    except Exception as e:
            print (e.message)
KellyGerrow
Esri Frequent Contributor

You can also try this sample (older, written in python 2.7) and remove the editor tracking bits if this isn't needed:

http://cloudygis.maps.arcgis.com/home/item.html?id=2d7027b148e24005916e55ab2cb74b50 

Also, try taking a look at the ArcGIS Python API

Add Item (upload FGDB): arcgis.gis module — arcgis 1.3.0 documentation 

Publish Item including overwrite: arcgis.gis module — arcgis 1.3.0 documentation 

-Kelly

0 Kudos
HaskettGeorge
New Contributor III

Thomas,

For clarification, will this code example replace the pre-existing data or append new data to it?

Also, what version of Python was your example written in?

I am looking to append data (new event data) to a pre-existing database in ArcGIS Online, similar to what you can do manually.  However, I want to automate the process.

Thanks,

0 Kudos
MarkVolz
Occasional Contributor III

Can we run this code without having to store our username and password in plain text?

0 Kudos
KellyGerrow
Esri Frequent Contributor

Sure,

You can modify how you add the username and password using any standard python method. In order to run the script above, you will need to somehow generate a token (requires username and password) and then append it to all of the required requests.

0 Kudos