Overwrite ArcGIS Online Feature Service using ArcGIS Pro Project

[#1]

Previously, I wrote an document on how to overwrite an ArcGIS Online feature service by referencing a feature class and using a truncate/append method.  I received a lot of feedback from this document, with some users encountering limitations such as attachments not being supported, and updating services containing multiple layers.  This solution is aimed to address these limitations.  Below is a script to overwrite an ArcGIS Online feature service by referencing an ArcGIS Pro project, and a video on how to use the script.  Please comment below if there are any issues or questions.

import arcpy, os, time
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection

# Variables
prjPath = r"C:\Projects\GeoNET\GeoNET.aprx"                 # Path to Pro Project
map = 'State Parks'                                         # Name of map in Pro Project
serviceDefID = '3fa1620c47dc490db43b9370e8cf5df8'           # Item ID of Service Definition
featureServiceID = 'fb42ef7b43154f95b8b6ad7357b7f663'       # Item ID of Feature Service
portal = "https://www.arcgis.com"                           # AGOL
user = "jskinner_rats"                                      # AGOL username
password = "********"                                       # AGOL password
shrOrg = False                                              # True/False to either share with Organization or not
shrEveryone = False                                         # True/False to either share with Everyone or not
shrGroups = ["f8e2c785c963441f9b2906e0cc269237"]            # Comma delimited Group IDs to share service with
preserveEditorTracking = True                               # True/False to preserve editor tracking from feature class
unregisterReplicas = True                                   # True/False to unregister existing replicas

# Set Environment Variables
arcpy.env.overwriteOutput = 1

# Start Timer
startTime = time.time()

print(f"Connecting to AGOL")
gis = GIS(portal, user, password)
arcpy.SignInToPortal(portal, user, password)

# Local paths to create temporary content
sddraft = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sddraft")
sd = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sd")
sdItem = gis.content.get(serviceDefID)

# Create a new SDDraft and stage to SD
print("Creating SD file")
arcpy.env.overwriteOutput = True
prj = arcpy.mp.ArcGISProject(prjPath)
mp = prj.listMaps(map)[0]
serviceDefName = sdItem.title
arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, serviceDefName, 'MY_HOSTED_SERVICES', 'FEATURE_ACCESS', '', True, True)
arcpy.StageService_server(sddraft, sd)

# Reference existing feature service to get properties
fsItem = gis.content.get(featureServiceID)
flyrCollection = FeatureLayerCollection.fromitem(fsItem)
existingDef = flyrCollection.properties

# Get thumbnail and metadata
thumbnail_file = fsItem.download_thumbnail(arcpy.env.scratchFolder)
metadata_file = fsItem.download_metadata(arcpy.env.scratchFolder)

# Unregister existing replicas
if unregisterReplicas:
    if flyrCollection.properties.syncEnabled:
        print("Unregister existing replicas")
        for replica in flyrCollection.replicas.get_list():
            replicaID = replica['replicaID']
            flyrCollection.replicas.unregister(replicaID)

# Overwrite feature service
sdItem.update(data=sd)
print("Overwriting existing feature service")
if preserveEditorTracking:
    pub_params = {"editorTrackingInfo" : {"preserveEditUsersAndTimestamps":'true'}}
    fs = sdItem.publish(overwrite=True, publish_parameters=pub_params)
else:
    fs = sdItem.publish(overwrite=True)

# Update service with previous properties
print("Updating service properties")
flyrCollection.manager.update_definition(existingDef)

# Update thumbnail and metadata
print("Updating thumbnail and metadata")
fs.update(thumbnail=thumbnail_file, metadata=metadata_file)

print("Setting sharing options")
fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)

print("Clearing scratch directory")
arcpy.env.workspace = arcpy.env.scratchFolder
for file in arcpy.ListFiles():
    if file.split(".")[-1] in ('sd', 'sddraft', 'png', 'xml'):
        arcpy.Delete_management(file)

endTime = time.time()
elapsedTime = round((endTime - startTime) / 60, 2)
print(f"Script completed in {elapsedTime} minutes")

Update 2/2/24:  Added the option to unregister existing replicas

[ModernElectric]

@JakeSkinner 

This version is working great. Been working with it all week on my electric and water dataset(s). All of my relationship classes stay intact and the attachments are all loaded after running the script. Will continue to work with it and develop my update workflow and let you know if I run into any new issues or bugs.

As of now, the Hosted Feature Layers are used in a web map being used in Field Maps. We are also using ESRI Workforce for our field crews and those maps/datasets need to have sync enabled, I am curious on adding those feature layers to some of our Workforce Projects and seeing how this update procedure works.

Appreciate your help!!

[KatGIS]

Hi @JakeSkinner,

Thank you so much for this updated script! 

It's exactly what we have been attempting to keep sync enabled and not break the replicas, and also overwrite the HFL with attachments.

Unfortunately I am running into an issue once testing it with an offline layer in Field Maps. The script runs perfectly before adding it to Field Maps (I believe before creating any replicas?). 

But in the "Overwrite Feature Service" block, I receive this error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
In  [65]:
Line 9:     fs = sdItem.publish(overwrite=True)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in publish:
Line 12740: elif not buildInitialCache and ret[0]["type"].lower() == "image service":

KeyError: 'type'
---------------------------------------------------------------------------

I removed the replica by removing it from Field Maps, and tested the script again and it still works.

We have field workers offline for days at a time, so this is a roadblock. 

The script is the same besides the fact I must sign into AGOL through Pro gis = GIS("pro") in order to bypass mandatory Multi-Factor Authentication.

Any idea how this error could be resolved?

Greatly appreciate your help! 🙂

[DJB]

@JakeSkinner 

I've been using very similar code to overwrite service on a nightly basis.  Up until this morning it has been working great.  I now get a warning that item.share has been deprecated and that I now need to use item.sharing.

DeprecatedWarning: share is deprecated as of 2.3.0 and has be removed in 3.0.0. Use `Item.sharing` instead.

I'm having a hard time following the documentation that describes the new sharing module.  

If you have any suggestions or material that outlines how to update the sharing to everyone I would greatly appreciate it.

I did find an example where the sddraft XML is altered but that seems like a step backwards and requires a lot more code for what use to take just a couple lines of code.

# Read the .sddraft file
docs = DOM.parse(sddraft_output_filename)
key_list = docs.getElementsByTagName('Key')
value_list = docs.getElementsByTagName('Value')

# Change following to "true" to share
SharetoOrganization = "false"
SharetoEveryone = "true"
SharetoGroup = "false"
# If SharetoGroup is set to "true", uncomment line below and provide group IDs
GroupID = ""    # GroupID = "f07fab920d71339cb7b1291e3059b7a8, e0fb8fff410b1d7bae1992700567f54a"

# Each key has a corresponding value. In all the cases, value of key_list[i] is value_list[i].
for i in range(key_list.length):
    if key_list[i].firstChild.nodeValue == "PackageUnderMyOrg":
        value_list[i].firstChild.nodeValue = SharetoOrganization
    if key_list[i].firstChild.nodeValue == "PackageIsPublic":
        value_list[i].firstChild.nodeValue = SharetoEveryone
    if key_list[i].firstChild.nodeValue == "PackageShareGroups":
        value_list[i].firstChild.nodeValue = SharetoGroup
    if SharetoGroup == "true" and key_list[i].firstChild.nodeValue == "PackageGroupIDs":
        value_list[i].firstChild.nodeValue = GroupID

This is how I use to do it.

# Set sharing options
shrOrg = True
shrEveryone = True
shrGroups = ""

if shrOrg or shrEveryone or shrGroups:
    print("Setting sharing options…")
    fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)

Thank you

[JakeSkinner]

@DJB you can update the sharing with the following:

from arcgis.gis._impl._content_manager import SharingLevel

sharing_mgr = fs.sharing
if shrOrg:
    sharing_mgr.sharing_level = SharingLevel.ORG
if shrEveryone:
    sharing_mgr.sharing_level = SharingLevel.EVERYONE
if shrGroups:
    for groupID in shrGroups:
        group = gis.groups.get(groupID)
        item_grp_sharing_mgr = sharing_mgr.groups
        item_grp_sharing_mgr.add(group=group)

I did notice if the sharing is already set on the feature service, you can omit this entirely and it will be maintained.  I can't recall if it use to do this or not at earlier versions of the API.

[DJB]

@JakeSkinner 

Thanks for the assistance Jake.  I wasn't aware that when overwriting a feature service it will still honour the original sharing properties.

I will definitely use this new code for when I need to alter sharing properties in the future.

Thanks again for your help Jake.  Cheers!