The article you are trying to access is permanently deleted.
Previously, I wrote an document on how to overwrite an ArcGIS Online feature service by referencing a feature class and using a truncate/append method. I received a lot of feedback from this document, with some users encountering limitations such as attachments not being supported, and updating services containing multiple layers. This solution is aimed to address these limitations. Below is a script to overwrite an ArcGIS Online feature service by referencing an ArcGIS Pro project, and a video on how to use the script. Please comment below if there are any issues or questions.
import arcpy, os, time, requests, json
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection
# Variables
prjPath = r"C:\Projects\GeoNET\GeoNET.aprx" # Path to Pro Project
map = 'State Parks' # Name of map in Pro Project
serviceDefID = '3fa1620c47dc490db43b9370e8cf5df8' # Item ID of Service Definition
featureServiceID = 'fb42ef7b43154f95b8b6ad7357b7f663' # Item ID of Feature Service
portal = "https://www.arcgis.com" # AGOL
user = "jskinner_rats" # AGOL username
password = "********" # AGOL password
preserveEditorTracking = True # True/False to preserve editor tracking from feature class
unregisterReplicas = True # True/False to unregister existing replicas
# Set Environment Variables
arcpy.env.overwriteOutput = 1
# Disable warnings
requests.packages.urllib3.disable_warnings()
# Start Timer
startTime = time.time()
print(f"Connecting to AGOL")
gis = GIS(portal, user, password)
arcpy.SignInToPortal(portal, user, password)
# Local paths to create temporary content
sddraft = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sddraft")
sd = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sd")
sdItem = gis.content.get(serviceDefID)
# Create a new SDDraft and stage to SD
print("Creating SD file")
arcpy.env.overwriteOutput = True
prj = arcpy.mp.ArcGISProject(prjPath)
mp = prj.listMaps(map)[0]
serviceDefName = sdItem.title
arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, serviceDefName, 'MY_HOSTED_SERVICES',
'FEATURE_ACCESS', '', True, True)
arcpy.StageService_server(sddraft, sd)
# Reference existing feature service to get properties
fsItem = gis.content.get(featureServiceID)
flyrCollection = FeatureLayerCollection.fromitem(fsItem)
properties = fsItem.get_data()
# Get thumbnail and metadata
thumbnail_file = fsItem.download_thumbnail(arcpy.env.scratchFolder)
metadata_file = fsItem.download_metadata(arcpy.env.scratchFolder)
# Unregister existing replicas
if unregisterReplicas:
if flyrCollection.properties.syncEnabled:
print("Unregister existing replicas")
for replica in flyrCollection.replicas.get_list():
replicaID = replica['replicaID']
flyrCollection.replicas.unregister(replicaID)
# Overwrite feature service
sdItem.update(data=sd)
print("Overwriting existing feature service")
if preserveEditorTracking:
pub_params = {"editorTrackingInfo" : {"preserveEditUsersAndTimestamps":'true'}}
fs = sdItem.publish(overwrite=True, publish_parameters=pub_params)
else:
fs = sdItem.publish(overwrite=True)
# Update service with previous properties
print("Updating service properties")
params = {'f': 'pjson', 'id': featureServiceID, 'text': json.dumps(properties), 'token': gis._con.token}
agolOrg = str(gis).split('@ ')[1].split(' ')[0]
postURL = f'{agolOrg}/sharing/rest/content/users/{fsItem.owner}/items/{featureServiceID}/update'
r = requests.post(postURL, data=params, verify=False)
response = json.loads(r.content)
# Update thumbnail and metadata
print("Updating thumbnail and metadata")
fs.update(thumbnail=thumbnail_file, metadata=metadata_file)
print("Clearing scratch directory")
arcpy.env.workspace = arcpy.env.scratchFolder
for file in arcpy.ListFiles():
if file.split(".")[-1] in ('sd', 'sddraft', 'png', 'xml'):
arcpy.Delete_management(file)
endTime = time.time()
elapsedTime = round((endTime - startTime) / 60, 2)
print(f"Script completed in {elapsedTime} minutes")
Update 2/2/24: Added the option to unregister existing replicas
This version is working great. Been working with it all week on my electric and water dataset(s). All of my relationship classes stay intact and the attachments are all loaded after running the script. Will continue to work with it and develop my update workflow and let you know if I run into any new issues or bugs.
As of now, the Hosted Feature Layers are used in a web map being used in Field Maps. We are also using ESRI Workforce for our field crews and those maps/datasets need to have sync enabled, I am curious on adding those feature layers to some of our Workforce Projects and seeing how this update procedure works.
Appreciate your help!!
Hi @JakeSkinner,
Thank you so much for this updated script!
It's exactly what we have been attempting to keep sync enabled and not break the replicas, and also overwrite the HFL with attachments.
Unfortunately I am running into an issue once testing it with an offline layer in Field Maps. The script runs perfectly before adding it to Field Maps (I believe before creating any replicas?).
But in the "Overwrite Feature Service" block, I receive this error:
--------------------------------------------------------------------------- KeyError Traceback (most recent call last) In [65]: Line 9: fs = sdItem.publish(overwrite=True) File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in publish: Line 12740: elif not buildInitialCache and ret[0]["type"].lower() == "image service": KeyError: 'type' ---------------------------------------------------------------------------
I removed the replica by removing it from Field Maps, and tested the script again and it still works.
We have field workers offline for days at a time, so this is a roadblock.
The script is the same besides the fact I must sign into AGOL through Pro gis = GIS("pro") in order to bypass mandatory Multi-Factor Authentication.
Any idea how this error could be resolved?
Greatly appreciate your help! 🙂
@JakeSkinner
I've been using very similar code to overwrite service on a nightly basis. Up until this morning it has been working great. I now get a warning that item.share has been deprecated and that I now need to use item.sharing.
DeprecatedWarning: share is deprecated as of 2.3.0 and has be removed in 3.0.0. Use `Item.sharing` instead.
I'm having a hard time following the documentation that describes the new sharing module.
If you have any suggestions or material that outlines how to update the sharing to everyone I would greatly appreciate it.
I did find an example where the sddraft XML is altered but that seems like a step backwards and requires a lot more code for what use to take just a couple lines of code.
# Read the .sddraft file
docs = DOM.parse(sddraft_output_filename)
key_list = docs.getElementsByTagName('Key')
value_list = docs.getElementsByTagName('Value')
# Change following to "true" to share
SharetoOrganization = "false"
SharetoEveryone = "true"
SharetoGroup = "false"
# If SharetoGroup is set to "true", uncomment line below and provide group IDs
GroupID = "" # GroupID = "f07fab920d71339cb7b1291e3059b7a8, e0fb8fff410b1d7bae1992700567f54a"
# Each key has a corresponding value. In all the cases, value of key_list[i] is value_list[i].
for i in range(key_list.length):
if key_list[i].firstChild.nodeValue == "PackageUnderMyOrg":
value_list[i].firstChild.nodeValue = SharetoOrganization
if key_list[i].firstChild.nodeValue == "PackageIsPublic":
value_list[i].firstChild.nodeValue = SharetoEveryone
if key_list[i].firstChild.nodeValue == "PackageShareGroups":
value_list[i].firstChild.nodeValue = SharetoGroup
if SharetoGroup == "true" and key_list[i].firstChild.nodeValue == "PackageGroupIDs":
value_list[i].firstChild.nodeValue = GroupID
This is how I use to do it.
# Set sharing options
shrOrg = True
shrEveryone = True
shrGroups = ""
if shrOrg or shrEveryone or shrGroups:
print("Setting sharing options…")
fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)
Thank you
@DJB you can update the sharing with the following:
from arcgis.gis._impl._content_manager import SharingLevel
sharing_mgr = fs.sharing
if shrOrg:
sharing_mgr.sharing_level = SharingLevel.ORG
if shrEveryone:
sharing_mgr.sharing_level = SharingLevel.EVERYONE
if shrGroups:
for groupID in shrGroups:
group = gis.groups.get(groupID)
item_grp_sharing_mgr = sharing_mgr.groups
item_grp_sharing_mgr.add(group=group)
I did notice if the sharing is already set on the feature service, you can omit this entirely and it will be maintained. I can't recall if it use to do this or not at earlier versions of the API.
Thanks for the assistance Jake. I wasn't aware that when overwriting a feature service it will still honour the original sharing properties.
I will definitely use this new code for when I need to alter sharing properties in the future.
Thanks again for your help Jake. Cheers!
Hi @JakeSkinner, thanks very much for sharing this. Unfortunately, when I run the script, my symbology and pop-up configurations (both having been defined in the Visualization tab) are not preserved. I believe that's happening because that info is stored at the portal item level, not at the service level, so they aren't captured in existingDef (they would have be to grabbed via fsItem.get_data()). But in your video, I see these properties are preserved. I can't figure how that's possible. What am I missing?
Hi @JakeSkinner thank you for this! I am not a programmer but I tried this and am getting what seems to be a sign-in to portal error. Do you know if anyone else has had this problem? I know I'm using the correct user name and password. I am using AGOL not Enterprise so this should be pretty straightforward I think. Thank you, Kristal
@KristalWalsh do you know if you are using a built-in AGOL account? Or, are you using a SAML user account? With SAML, there is an icon you can click that will sign you in using the same account you typically sign into Windows with.
@JakeSkinner hi, thank you, I have an organizational account for a government agency so not sure if that is considered "built-in". I am signed in to my AGO account in another window not that it makes any difference.
@KristalWalsh is there an '@' symbol in your username?
@JakeSkinner no, not in the body of the code where my user name is inserted. I followed your video exactly.
@JakeSkinner I noticed earlier that I was not signed in to Pro. I thought maybe I got disconnected at some point, so I just ran it again after signing in to Pro. It seems that when I execute the script, it signs me out of Pro. Should that happen?
@cjenkins_rva I'm not sure if something changed in the API or not, but I swore this was working before. However, I updated the code to use the .get_data() and apply this to the service using the requests module. I couldn't find a way via the API to apply the properties all at once for all layers; that functionality may not exist, yet. Anyways, give the updated code a try.
Been using the latest script to rebuild my update workflow.
On one of my Web Maps/Hosted Feature Layers - using this updated script - why am I getting:
mp = prj.listMaps(map)(0)
IndexError: list index out of range
@ModernElectric do you have the name of the map in ArcGIS Pro specified correctly for the map variable?
Disregard.
Please forgive my ignorance, lack of proof-reading abilities 😉
Updated Script is working perfectly for updating our AGOL dataset.
This script seems to be just what I am looking for, but I get an error when trying to login to AGOL.
Connecting to AGOL
Traceback (most recent call last):
File "E:\Scripts\Workorders_CD\NewOverwriteWebLayers.py", line 30, in <module>
arcpy.SignInToPortal(portal, user, password)
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\__init__.py", line 2609, in SignInToPortal
return _SignInToPortal(*args, **kwargs)
ValueError: Error signing on to https://arcgis.com/.
Message : s
Details : Unable to generate token.
Any Thoughts?
My post keeps getting marked as spam, not sure why. I'll try again. This script is just what I was looking for, but I am getting an error trying to log into AGOL.
Connecting to AGOL
Traceback (most recent call last):
File "E:\Scripts\Workorders_CD\NewOverwriteWebLayers.py", line 30, in <module>
arcpy.SignInToPortal(portal, user, password)
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\__init__.py", line 2609, in SignInToPortal
return _SignInToPortal(*args, **kwargs)
ValueError: Error signing on to https://arcgis.com/.
Message : s
Details : Unable to generate token.
Any ideas?
@cog_GIS_Admin check the casing of your username, this is case sensitive. For example, if your AGOL login is cog_GIS_Admin, and you specify cog_gis_admin, it will not authenticate.....even though cog_gis_admin will work in a web browser.
Thanks Jake, that was the issue. Now working on getting it to auto run with the server task scheduler.
@cog_GIS_Admin here is helpful document on how use Windows Task Scheduler with python scripts: