Select to view content in your preferred language

Overwrite ArcGIS Online Feature Service using ArcGIS Pro Project

1725
12
01-17-2024 12:38 PM

Overwrite ArcGIS Online Feature Service using ArcGIS Pro Project

Previously, I wrote an document on how to overwrite an ArcGIS Online feature service by referencing a feature class and using a truncate/append method.  I received a lot of feedback from this document, with some users encountering limitations such as attachments not being supported, and updating services containing multiple layers.  This solution is aimed to address these limitations.  Below is a script to overwrite an ArcGIS Online feature service by referencing an ArcGIS Pro project, and a video on how to use the script.  Please comment below if there are any issues or questions.

 

 

 

import arcpy, os, time
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection

# Variables
prjPath = r"C:\Projects\GeoNET\GeoNET.aprx"                 # Path to Pro Project
map = 'State Parks'                                         # Name of map in Pro Project
serviceDefID = '3fa1620c47dc490db43b9370e8cf5df8'           # Item ID of Service Definition
featureServiceID = 'fb42ef7b43154f95b8b6ad7357b7f663'       # Item ID of Feature Service
portal = "https://www.arcgis.com"                           # AGOL
user = "jskinner_rats"                                      # AGOL username
password = "********"                                       # AGOL password
shrOrg = False                                              # True/False to either share with Organization or not
shrEveryone = False                                         # True/False to either share with Everyone or not
shrGroups = ["f8e2c785c963441f9b2906e0cc269237"]            # Comma delimited Group IDs to share service with
preserveEditorTracking = True                               # True/False to preserve editor tracking from feature class
unregisterReplicas = True                                   # True/False to unregister existing replicas

# Set Environment Variables
arcpy.env.overwriteOutput = 1

# Start Timer
startTime = time.time()

print(f"Connecting to AGOL")
gis = GIS(portal, user, password)
arcpy.SignInToPortal(portal, user, password)

# Local paths to create temporary content
sddraft = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sddraft")
sd = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sd")
sdItem = gis.content.get(serviceDefID)

# Create a new SDDraft and stage to SD
print("Creating SD file")
arcpy.env.overwriteOutput = True
prj = arcpy.mp.ArcGISProject(prjPath)
mp = prj.listMaps(map)[0]
serviceDefName = sdItem.title
arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, serviceDefName, 'MY_HOSTED_SERVICES', 'FEATURE_ACCESS', '', True, True)
arcpy.StageService_server(sddraft, sd)

# Reference existing feature service to get properties
fsItem = gis.content.get(featureServiceID)
flyrCollection = FeatureLayerCollection.fromitem(fsItem)
existingDef = flyrCollection.properties

# Get thumbnail and metadata
thumbnail_file = fsItem.download_thumbnail(arcpy.env.scratchFolder)
metadata_file = fsItem.download_metadata(arcpy.env.scratchFolder)

# Unregister existing replicas
if unregisterReplicas:
    if flyrCollection.properties.syncEnabled:
        print("Unregister existing replicas")
        for replica in flyrCollection.replicas.get_list():
            replicaID = replica['replicaID']
            flyrCollection.replicas.unregister(replicaID)

# Overwrite feature service
sdItem.update(data=sd)
print("Overwriting existing feature service")
if preserveEditorTracking:
    pub_params = {"editorTrackingInfo" : {"preserveEditUsersAndTimestamps":'true'}}
    fs = sdItem.publish(overwrite=True, publish_parameters=pub_params)
else:
    fs = sdItem.publish(overwrite=True)

# Update service with previous properties
print("Updating service properties")
flyrCollection.manager.update_definition(existingDef)

# Update thumbnail and metadata
print("Updating thumbnail and metadata")
fs.update(thumbnail=thumbnail_file, metadata=metadata_file)

print("Setting sharing options")
fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)

print("Clearing scratch directory")
arcpy.env.workspace = arcpy.env.scratchFolder
for file in arcpy.ListFiles():
    if file.split(".")[-1] in ('sd', 'sddraft', 'png', 'xml'):
        arcpy.Delete_management(file)

endTime = time.time()
elapsedTime = round((endTime - startTime) / 60, 2)
print(f"Script completed in {elapsedTime} minutes")

 

 

 

Update 2/2/24:  Added the option to unregister existing replicas

Attachments
Comments
ModernElectric
Regular Contributor

@JakeSkinner 

This version is working great. Been working with it all week on my electric and water dataset(s). All of my relationship classes stay intact and the attachments are all loaded after running the script. Will continue to work with it and develop my update workflow and let you know if I run into any new issues or bugs.

As of now, the Hosted Feature Layers are used in a web map being used in Field Maps. We are also using ESRI Workforce for our field crews and those maps/datasets need to have sync enabled, I am curious on adding those feature layers to some of our Workforce Projects and seeing how this update procedure works.

Appreciate your help!!

KatGIS
by
New Contributor II

Hi @JakeSkinner,

Thank you so much for this updated script! 

It's exactly what we have been attempting to keep sync enabled and not break the replicas, and also overwrite the HFL with attachments.

Unfortunately I am running into an issue once testing it with an offline layer in Field Maps. The script runs perfectly before adding it to Field Maps (I believe before creating any replicas?). 

But in the "Overwrite Feature Service" block, I receive this error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
In  [65]:
Line 9:     fs = sdItem.publish(overwrite=True)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in publish:
Line 12740: elif not buildInitialCache and ret[0]["type"].lower() == "image service":

KeyError: 'type'
---------------------------------------------------------------------------

I removed the replica by removing it from Field Maps, and tested the script again and it still works.

We have field workers offline for days at a time, so this is a roadblock. 

The script is the same besides the fact I must sign into AGOL through Pro gis = GIS("pro") in order to bypass mandatory Multi-Factor Authentication.

Any idea how this error could be resolved?

Greatly appreciate your help! 🙂

 

DJB
by
New Contributor III

@JakeSkinner 

I've been using very similar code to overwrite service on a nightly basis.  Up until this morning it has been working great.  I now get a warning that item.share has been deprecated and that I now need to use item.sharing.

DeprecatedWarning: share is deprecated as of 2.3.0 and has be removed in 3.0.0. Use `Item.sharing` instead.

I'm having a hard time following the documentation that describes the new sharing module.  

If you have any suggestions or material that outlines how to update the sharing to everyone I would greatly appreciate it.

I did find an example where the sddraft XML is altered but that seems like a step backwards and requires a lot more code for what use to take just a couple lines of code.

# Read the .sddraft file
docs = DOM.parse(sddraft_output_filename)
key_list = docs.getElementsByTagName('Key')
value_list = docs.getElementsByTagName('Value')

# Change following to "true" to share
SharetoOrganization = "false"
SharetoEveryone = "true"
SharetoGroup = "false"
# If SharetoGroup is set to "true", uncomment line below and provide group IDs
GroupID = ""    # GroupID = "f07fab920d71339cb7b1291e3059b7a8, e0fb8fff410b1d7bae1992700567f54a"

# Each key has a corresponding value. In all the cases, value of key_list[i] is value_list[i].
for i in range(key_list.length):
    if key_list[i].firstChild.nodeValue == "PackageUnderMyOrg":
        value_list[i].firstChild.nodeValue = SharetoOrganization
    if key_list[i].firstChild.nodeValue == "PackageIsPublic":
        value_list[i].firstChild.nodeValue = SharetoEveryone
    if key_list[i].firstChild.nodeValue == "PackageShareGroups":
        value_list[i].firstChild.nodeValue = SharetoGroup
    if SharetoGroup == "true" and key_list[i].firstChild.nodeValue == "PackageGroupIDs":
        value_list[i].firstChild.nodeValue = GroupID

This is how I use to do it.

# Set sharing options
shrOrg = True
shrEveryone = True
shrGroups = ""

if shrOrg or shrEveryone or shrGroups:
    print("Setting sharing options…")
    fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)

 

Thank you

JakeSkinner
Esri Esteemed Contributor

@DJB you can update the sharing with the following:

from arcgis.gis._impl._content_manager import SharingLevel

sharing_mgr = fs.sharing
if shrOrg:
    sharing_mgr.sharing_level = SharingLevel.ORG
if shrEveryone:
    sharing_mgr.sharing_level = SharingLevel.EVERYONE
if shrGroups:
    for groupID in shrGroups:
        group = gis.groups.get(groupID)
        item_grp_sharing_mgr = sharing_mgr.groups
        item_grp_sharing_mgr.add(group=group)

I did notice if the sharing is already set on the feature service, you can omit this entirely and it will be maintained.  I can't recall if it use to do this or not at earlier versions of the API.

DJB
by
New Contributor III

@JakeSkinner 

Thanks for the assistance Jake.  I wasn't aware that when overwriting a feature service it will still honour the original sharing properties.

I will definitely use this new code for when I need to alter sharing properties in the future.

Thanks again for your help Jake.  Cheers!

cjenkins_rva
New Contributor

Hi @JakeSkinner, thanks very much for sharing this. Unfortunately, when I run the script, my symbology and pop-up configurations (both having been defined in the Visualization tab) are not preserved. I believe that's happening because that info is stored at the portal item level, not at the service level, so they aren't captured in existingDef (they would have be to grabbed via fsItem.get_data()). But in your video, I see these properties are preserved. I can't figure how that's possible. What am I missing?

KristalWalsh
Occasional Contributor II

Hi @JakeSkinner thank you for this! I am not a programmer but I tried this and am getting what seems to be a sign-in to portal error. Do you know if anyone else has had this problem? I know I'm using the correct user name and password. I am using AGOL not Enterprise so this should be pretty straightforward I think. Thank you, Kristal

JakeSkinner
Esri Esteemed Contributor

@KristalWalsh do you know if you are using a built-in AGOL account?  Or, are you using a SAML user account?  With SAML, there is an icon you can click that will sign you in using the same account you typically sign into Windows with.

KristalWalsh
Occasional Contributor II

@JakeSkinner hi, thank you, I have an organizational account for a government agency so not sure if that is considered "built-in". I am signed in to my AGO account in another window not that it makes any difference. 

JakeSkinner
Esri Esteemed Contributor

@KristalWalsh is there an '@' symbol in your username?

KristalWalsh
Occasional Contributor II

@JakeSkinner no, not in the body of the code where my user name is inserted. I followed your video exactly.

KristalWalsh
Occasional Contributor II

@JakeSkinner I noticed earlier that I was not signed in to Pro. I thought maybe I got disconnected at some point, so I just ran it again after signing in to Pro. It seems that when I execute the script, it signs me out of Pro. Should that happen?

Version history
Last update:
‎05-14-2024 04:14 PM
Updated by:
Contributors