Select to view content in your preferred language

Overwrite ArcGIS Online Feature Service using ArcGIS Pro Project

3010
21
01-17-2024 12:38 PM

Overwrite ArcGIS Online Feature Service using ArcGIS Pro Project

Previously, I wrote an document on how to overwrite an ArcGIS Online feature service by referencing a feature class and using a truncate/append method.  I received a lot of feedback from this document, with some users encountering limitations such as attachments not being supported, and updating services containing multiple layers.  This solution is aimed to address these limitations.  Below is a script to overwrite an ArcGIS Online feature service by referencing an ArcGIS Pro project, and a video on how to use the script.  Please comment below if there are any issues or questions.

 

 

import arcpy, os, time, requests, json
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection

# Variables
prjPath = r"C:\Projects\GeoNET\GeoNET.aprx"                 # Path to Pro Project
map = 'State Parks'                                         # Name of map in Pro Project
serviceDefID = '3fa1620c47dc490db43b9370e8cf5df8'           # Item ID of Service Definition
featureServiceID = 'fb42ef7b43154f95b8b6ad7357b7f663'       # Item ID of Feature Service
portal = "https://www.arcgis.com"                           # AGOL
user = "jskinner_rats"                                      # AGOL username
password = "********"                                       # AGOL password
preserveEditorTracking = True                               # True/False to preserve editor tracking from feature class
unregisterReplicas = True                                   # True/False to unregister existing replicas

# Set Environment Variables
arcpy.env.overwriteOutput = 1

# Disable warnings
requests.packages.urllib3.disable_warnings()

# Start Timer
startTime = time.time()

print(f"Connecting to AGOL")
gis = GIS(portal, user, password)
arcpy.SignInToPortal(portal, user, password)

# Local paths to create temporary content
sddraft = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sddraft")
sd = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sd")
sdItem = gis.content.get(serviceDefID)

# Create a new SDDraft and stage to SD
print("Creating SD file")
arcpy.env.overwriteOutput = True
prj = arcpy.mp.ArcGISProject(prjPath)
mp = prj.listMaps(map)[0]
serviceDefName = sdItem.title
arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, serviceDefName, 'MY_HOSTED_SERVICES',
                               'FEATURE_ACCESS', '', True, True)
arcpy.StageService_server(sddraft, sd)

# Reference existing feature service to get properties
fsItem = gis.content.get(featureServiceID)
flyrCollection = FeatureLayerCollection.fromitem(fsItem)
properties = fsItem.get_data()

# Get thumbnail and metadata
thumbnail_file = fsItem.download_thumbnail(arcpy.env.scratchFolder)
metadata_file = fsItem.download_metadata(arcpy.env.scratchFolder)

# Unregister existing replicas
if unregisterReplicas:
    if flyrCollection.properties.syncEnabled:
        print("Unregister existing replicas")
        for replica in flyrCollection.replicas.get_list():
            replicaID = replica['replicaID']
            flyrCollection.replicas.unregister(replicaID)

# Overwrite feature service
sdItem.update(data=sd)
print("Overwriting existing feature service")
if preserveEditorTracking:
    pub_params = {"editorTrackingInfo" : {"preserveEditUsersAndTimestamps":'true'}}
    fs = sdItem.publish(overwrite=True, publish_parameters=pub_params)
else:
    fs = sdItem.publish(overwrite=True)

# Update service with previous properties
print("Updating service properties")
params = {'f': 'pjson', 'id': featureServiceID, 'text': json.dumps(properties), 'token': gis._con.token}
agolOrg = str(gis).split('@ ')[1].split(' ')[0]
postURL = f'{agolOrg}/sharing/rest/content/users/{fsItem.owner}/items/{featureServiceID}/update'
r = requests.post(postURL, data=params, verify=False)
response = json.loads(r.content)

# Update thumbnail and metadata
print("Updating thumbnail and metadata")
fs.update(thumbnail=thumbnail_file, metadata=metadata_file)

print("Clearing scratch directory")
arcpy.env.workspace = arcpy.env.scratchFolder
for file in arcpy.ListFiles():
    if file.split(".")[-1] in ('sd', 'sddraft', 'png', 'xml'):
        arcpy.Delete_management(file)

endTime = time.time()
elapsedTime = round((endTime - startTime) / 60, 2)
print(f"Script completed in {elapsedTime} minutes")

 

 

 

 

Update 2/2/24:  Added the option to unregister existing replicas

Attachments
Comments
ModernElectric
Frequent Contributor

@JakeSkinner 

This version is working great. Been working with it all week on my electric and water dataset(s). All of my relationship classes stay intact and the attachments are all loaded after running the script. Will continue to work with it and develop my update workflow and let you know if I run into any new issues or bugs.

As of now, the Hosted Feature Layers are used in a web map being used in Field Maps. We are also using ESRI Workforce for our field crews and those maps/datasets need to have sync enabled, I am curious on adding those feature layers to some of our Workforce Projects and seeing how this update procedure works.

Appreciate your help!!

KatGIS
by
Occasional Contributor

Hi @JakeSkinner,

Thank you so much for this updated script! 

It's exactly what we have been attempting to keep sync enabled and not break the replicas, and also overwrite the HFL with attachments.

Unfortunately I am running into an issue once testing it with an offline layer in Field Maps. The script runs perfectly before adding it to Field Maps (I believe before creating any replicas?). 

But in the "Overwrite Feature Service" block, I receive this error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
In  [65]:
Line 9:     fs = sdItem.publish(overwrite=True)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in publish:
Line 12740: elif not buildInitialCache and ret[0]["type"].lower() == "image service":

KeyError: 'type'
---------------------------------------------------------------------------

I removed the replica by removing it from Field Maps, and tested the script again and it still works.

We have field workers offline for days at a time, so this is a roadblock. 

The script is the same besides the fact I must sign into AGOL through Pro gis = GIS("pro") in order to bypass mandatory Multi-Factor Authentication.

Any idea how this error could be resolved?

Greatly appreciate your help! 🙂

 

DJB
by
Regular Contributor

@JakeSkinner 

I've been using very similar code to overwrite service on a nightly basis.  Up until this morning it has been working great.  I now get a warning that item.share has been deprecated and that I now need to use item.sharing.

DeprecatedWarning: share is deprecated as of 2.3.0 and has be removed in 3.0.0. Use `Item.sharing` instead.

I'm having a hard time following the documentation that describes the new sharing module.  

If you have any suggestions or material that outlines how to update the sharing to everyone I would greatly appreciate it.

I did find an example where the sddraft XML is altered but that seems like a step backwards and requires a lot more code for what use to take just a couple lines of code.

# Read the .sddraft file
docs = DOM.parse(sddraft_output_filename)
key_list = docs.getElementsByTagName('Key')
value_list = docs.getElementsByTagName('Value')

# Change following to "true" to share
SharetoOrganization = "false"
SharetoEveryone = "true"
SharetoGroup = "false"
# If SharetoGroup is set to "true", uncomment line below and provide group IDs
GroupID = ""    # GroupID = "f07fab920d71339cb7b1291e3059b7a8, e0fb8fff410b1d7bae1992700567f54a"

# Each key has a corresponding value. In all the cases, value of key_list[i] is value_list[i].
for i in range(key_list.length):
    if key_list[i].firstChild.nodeValue == "PackageUnderMyOrg":
        value_list[i].firstChild.nodeValue = SharetoOrganization
    if key_list[i].firstChild.nodeValue == "PackageIsPublic":
        value_list[i].firstChild.nodeValue = SharetoEveryone
    if key_list[i].firstChild.nodeValue == "PackageShareGroups":
        value_list[i].firstChild.nodeValue = SharetoGroup
    if SharetoGroup == "true" and key_list[i].firstChild.nodeValue == "PackageGroupIDs":
        value_list[i].firstChild.nodeValue = GroupID

This is how I use to do it.

# Set sharing options
shrOrg = True
shrEveryone = True
shrGroups = ""

if shrOrg or shrEveryone or shrGroups:
    print("Setting sharing options…")
    fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)

 

Thank you

JakeSkinner
Esri Esteemed Contributor

@DJB you can update the sharing with the following:

from arcgis.gis._impl._content_manager import SharingLevel

sharing_mgr = fs.sharing
if shrOrg:
    sharing_mgr.sharing_level = SharingLevel.ORG
if shrEveryone:
    sharing_mgr.sharing_level = SharingLevel.EVERYONE
if shrGroups:
    for groupID in shrGroups:
        group = gis.groups.get(groupID)
        item_grp_sharing_mgr = sharing_mgr.groups
        item_grp_sharing_mgr.add(group=group)

I did notice if the sharing is already set on the feature service, you can omit this entirely and it will be maintained.  I can't recall if it use to do this or not at earlier versions of the API.

DJB
by
Regular Contributor

@JakeSkinner 

Thanks for the assistance Jake.  I wasn't aware that when overwriting a feature service it will still honour the original sharing properties.

I will definitely use this new code for when I need to alter sharing properties in the future.

Thanks again for your help Jake.  Cheers!

cjenkins_rva
Frequent Explorer

Hi @JakeSkinner, thanks very much for sharing this. Unfortunately, when I run the script, my symbology and pop-up configurations (both having been defined in the Visualization tab) are not preserved. I believe that's happening because that info is stored at the portal item level, not at the service level, so they aren't captured in existingDef (they would have be to grabbed via fsItem.get_data()). But in your video, I see these properties are preserved. I can't figure how that's possible. What am I missing?

KristalWalsh
Frequent Contributor

Hi @JakeSkinner thank you for this! I am not a programmer but I tried this and am getting what seems to be a sign-in to portal error. Do you know if anyone else has had this problem? I know I'm using the correct user name and password. I am using AGOL not Enterprise so this should be pretty straightforward I think. Thank you, Kristal

JakeSkinner
Esri Esteemed Contributor

@KristalWalsh do you know if you are using a built-in AGOL account?  Or, are you using a SAML user account?  With SAML, there is an icon you can click that will sign you in using the same account you typically sign into Windows with.

KristalWalsh
Frequent Contributor

@JakeSkinner hi, thank you, I have an organizational account for a government agency so not sure if that is considered "built-in". I am signed in to my AGO account in another window not that it makes any difference. 

JakeSkinner
Esri Esteemed Contributor

@KristalWalsh is there an '@' symbol in your username?

KristalWalsh
Frequent Contributor

@JakeSkinner no, not in the body of the code where my user name is inserted. I followed your video exactly.

KristalWalsh
Frequent Contributor

@JakeSkinner I noticed earlier that I was not signed in to Pro. I thought maybe I got disconnected at some point, so I just ran it again after signing in to Pro. It seems that when I execute the script, it signs me out of Pro. Should that happen?

JakeSkinner
Esri Esteemed Contributor

@cjenkins_rva I'm not sure if something changed in the API or not, but I swore this was working before.  However, I updated the code to use the .get_data() and apply this to the service using the requests module.  I couldn't find a way via the API to apply the properties all at once for all layers; that functionality may not exist, yet.  Anyways, give the updated code a try.

ModernElectric
Frequent Contributor

@JakeSkinner 

Been using the latest script to rebuild my update workflow.

On one of my Web Maps/Hosted Feature Layers - using this updated script - why am I getting:

mp = prj.listMaps(map)(0)

IndexError: list index out of range

 

JakeSkinner
Esri Esteemed Contributor

@ModernElectric do you have the name of the map in ArcGIS Pro specified correctly for the map variable?

JakeSkinner_0-1726665486634.png

 

ModernElectric
Frequent Contributor

Disregard.

Please forgive my ignorance, lack of proof-reading abilities 😉 

Updated Script is working perfectly for updating our AGOL dataset.

cog_GIS_Admin
Emerging Contributor

This script seems to be just what I am looking for, but I get an error when trying to login to AGOL.

Connecting to AGOL
Traceback (most recent call last):
File "E:\Scripts\Workorders_CD\NewOverwriteWebLayers.py", line 30, in <module>
arcpy.SignInToPortal(portal, user, password)
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\__init__.py", line 2609, in SignInToPortal
return _SignInToPortal(*args, **kwargs)
ValueError: Error signing on to https://arcgis.com/.

Message : s
Details : Unable to generate token.

 

Any Thoughts?

cog_GIS_Admin
Emerging Contributor

My post keeps getting marked as spam, not sure why. I'll try again. This script is just what I was looking for, but I am getting an error trying to log into AGOL.

 

Connecting to AGOL
Traceback (most recent call last):
File "E:\Scripts\Workorders_CD\NewOverwriteWebLayers.py", line 30, in <module>
arcpy.SignInToPortal(portal, user, password)
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\__init__.py", line 2609, in SignInToPortal
return _SignInToPortal(*args, **kwargs)
ValueError: Error signing on to https://arcgis.com/.

Message : s
Details : Unable to generate token.

 

Any ideas?

 

JakeSkinner
Esri Esteemed Contributor

@cog_GIS_Admin check the casing of your username, this is case sensitive.  For example, if your AGOL login is cog_GIS_Admin, and you specify cog_gis_admin, it will not authenticate.....even though cog_gis_admin will work in a web browser.

cog_GIS_Admin
Emerging Contributor

Thanks Jake, that was the issue. Now working on getting it to auto run with the server task scheduler.

JakeSkinner
Esri Esteemed Contributor

@cog_GIS_Admin here is helpful document on how use Windows Task Scheduler with python scripts:

https://community.esri.com/t5/python-documents/schedule-a-python-script-using-windows-task/ta-p/9158...

Version history
Last update:
‎09-12-2024 08:24 AM
Updated by:
Contributors