Select to view content in your preferred language

Overwrite ArcGIS Online Feature Service using ArcGIS Pro Project

6260
35
01-17-2024 12:38 PM

Overwrite ArcGIS Online Feature Service using ArcGIS Pro Project

Previously, I wrote an document on how to overwrite an ArcGIS Online feature service by referencing a feature class and using a truncate/append method.  I received a lot of feedback from this document, with some users encountering limitations such as attachments not being supported, and updating services containing multiple layers.  This solution is aimed to address these limitations.  Below is a script to overwrite an ArcGIS Online feature service by referencing an ArcGIS Pro project, and a video on how to use the script.  Please comment below if there are any issues or questions.

 

 

 

 

import arcpy, os, time, requests, json
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection

# Variables
prjPath = r"C:\Projects\GeoNET\GeoNET.aprx"                 # Path to Pro Project
map = 'State Parks'                                         # Name of map in Pro Project
serviceDefID = '3fa1620c47dc490db43b9370e8cf5df8'           # Item ID of Service Definition
featureServiceID = 'fb42ef7b43154f95b8b6ad7357b7f663'       # Item ID of Feature Service
portal = "https://www.arcgis.com"                           # AGOL
user = "jskinner_rats"                                      # AGOL username
password = "********"                                       # AGOL password
preserveEditorTracking = True                               # True/False to preserve editor tracking from feature class
unregisterReplicas = True                                   # True/False to unregister existing replicas

# Set Environment Variables
arcpy.env.overwriteOutput = 1

# Disable warnings
requests.packages.urllib3.disable_warnings()

# Start Timer
startTime = time.time()

print(f"Connecting to AGOL")
gis = GIS(portal, user, password)
arcpy.SignInToPortal(portal, user, password)

# Local paths to create temporary content
sddraft = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sddraft")
sd = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sd")
sdItem = gis.content.get(serviceDefID)

# Create a new SDDraft and stage to SD
print("Creating SD file")
arcpy.env.overwriteOutput = True
prj = arcpy.mp.ArcGISProject(prjPath)
mp = prj.listMaps(map)[0]
serviceDefName = sdItem.title
arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, serviceDefName, 'MY_HOSTED_SERVICES',
                               'FEATURE_ACCESS', '', True, True)
arcpy.StageService_server(sddraft, sd)

# Reference existing feature service to get properties
fsItem = gis.content.get(featureServiceID)
flyrCollection = FeatureLayerCollection.fromitem(fsItem)
properties = fsItem.get_data()

# Get thumbnail and metadata
thumbnail_file = fsItem.download_thumbnail(arcpy.env.scratchFolder)
metadata_file = fsItem.download_metadata(arcpy.env.scratchFolder)

# Unregister existing replicas
enableSync = False
if unregisterReplicas:
    if flyrCollection.properties.syncEnabled:
        enableSync = True
        print("Unregister existing replicas")
        for replica in flyrCollection.replicas.get_list():
            replicaID = replica['replicaID']
            flyrCollection.replicas.unregister(replicaID)

# Overwrite feature service
sdItem.update(data=sd)
print("Overwriting existing feature service")
if preserveEditorTracking:
    pub_params = {"editorTrackingInfo" : {"preserveEditUsersAndTimestamps":'true'}}
    fs = sdItem.publish(overwrite=True, publish_parameters=pub_params)
else:
    fs = sdItem.publish(overwrite=True)

# Update service with previous properties
print("Updating service properties")
params = {'f': 'pjson', 'id': featureServiceID, 'text': json.dumps(properties), 'token': gis._con.token}
agolOrg = str(gis).split('@ ')[1].split(' ')[0]
postURL = f'{agolOrg}/sharing/rest/content/users/{fsItem.owner}/items/{featureServiceID}/update'
r = requests.post(postURL, data=params, verify=False)
response = json.loads(r.content)

# Enable sync
if enableSync:
    properties = flyrCollection.properties.capabilities
    updateDict = {"capabilities": "Query", "syncEnabled": False}
    flyrCollection.manager.update_definition(updateDict)
    print("Enabling Sync")
    updateDict = {"capabilities": properties, "syncEnabled": True}
    flyrCollection.manager.update_definition(updateDict)

# Update thumbnail and metadata
print("Updating thumbnail and metadata")
fs.update(thumbnail=thumbnail_file, metadata=metadata_file)

print("Clearing scratch directory")
arcpy.env.workspace = arcpy.env.scratchFolder
for file in arcpy.ListFiles():
    if file.split(".")[-1] in ('sd', 'sddraft', 'png', 'xml'):
        arcpy.Delete_management(file)

endTime = time.time()
elapsedTime = round((endTime - startTime) / 60, 2)
print(f"Script completed in {elapsedTime} minutes")

 

 

 

Updates

Update 2/2/24:  Added the option to unregister existing replicas

Update 12/2/24:  Sync is re-enabled if it was previously enabled

Attachments
Comments
ModernElectric
Frequent Contributor

@JakeSkinner 

This version is working great. Been working with it all week on my electric and water dataset(s). All of my relationship classes stay intact and the attachments are all loaded after running the script. Will continue to work with it and develop my update workflow and let you know if I run into any new issues or bugs.

As of now, the Hosted Feature Layers are used in a web map being used in Field Maps. We are also using ESRI Workforce for our field crews and those maps/datasets need to have sync enabled, I am curious on adding those feature layers to some of our Workforce Projects and seeing how this update procedure works.

Appreciate your help!!

KatGIS
by
Occasional Contributor

Hi @JakeSkinner,

Thank you so much for this updated script! 

It's exactly what we have been attempting to keep sync enabled and not break the replicas, and also overwrite the HFL with attachments.

Unfortunately I am running into an issue once testing it with an offline layer in Field Maps. The script runs perfectly before adding it to Field Maps (I believe before creating any replicas?). 

But in the "Overwrite Feature Service" block, I receive this error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
In  [65]:
Line 9:     fs = sdItem.publish(overwrite=True)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in publish:
Line 12740: elif not buildInitialCache and ret[0]["type"].lower() == "image service":

KeyError: 'type'
---------------------------------------------------------------------------

I removed the replica by removing it from Field Maps, and tested the script again and it still works.

We have field workers offline for days at a time, so this is a roadblock. 

The script is the same besides the fact I must sign into AGOL through Pro gis = GIS("pro") in order to bypass mandatory Multi-Factor Authentication.

Any idea how this error could be resolved?

Greatly appreciate your help! 🙂

 

DJB
by
Regular Contributor

@JakeSkinner 

I've been using very similar code to overwrite service on a nightly basis.  Up until this morning it has been working great.  I now get a warning that item.share has been deprecated and that I now need to use item.sharing.

DeprecatedWarning: share is deprecated as of 2.3.0 and has be removed in 3.0.0. Use `Item.sharing` instead.

I'm having a hard time following the documentation that describes the new sharing module.  

If you have any suggestions or material that outlines how to update the sharing to everyone I would greatly appreciate it.

I did find an example where the sddraft XML is altered but that seems like a step backwards and requires a lot more code for what use to take just a couple lines of code.

# Read the .sddraft file
docs = DOM.parse(sddraft_output_filename)
key_list = docs.getElementsByTagName('Key')
value_list = docs.getElementsByTagName('Value')

# Change following to "true" to share
SharetoOrganization = "false"
SharetoEveryone = "true"
SharetoGroup = "false"
# If SharetoGroup is set to "true", uncomment line below and provide group IDs
GroupID = ""    # GroupID = "f07fab920d71339cb7b1291e3059b7a8, e0fb8fff410b1d7bae1992700567f54a"

# Each key has a corresponding value. In all the cases, value of key_list[i] is value_list[i].
for i in range(key_list.length):
    if key_list[i].firstChild.nodeValue == "PackageUnderMyOrg":
        value_list[i].firstChild.nodeValue = SharetoOrganization
    if key_list[i].firstChild.nodeValue == "PackageIsPublic":
        value_list[i].firstChild.nodeValue = SharetoEveryone
    if key_list[i].firstChild.nodeValue == "PackageShareGroups":
        value_list[i].firstChild.nodeValue = SharetoGroup
    if SharetoGroup == "true" and key_list[i].firstChild.nodeValue == "PackageGroupIDs":
        value_list[i].firstChild.nodeValue = GroupID

This is how I use to do it.

# Set sharing options
shrOrg = True
shrEveryone = True
shrGroups = ""

if shrOrg or shrEveryone or shrGroups:
    print("Setting sharing options…")
    fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)

 

Thank you

JakeSkinner
Esri Esteemed Contributor

@DJB you can update the sharing with the following:

from arcgis.gis._impl._content_manager import SharingLevel

sharing_mgr = fs.sharing
if shrOrg:
    sharing_mgr.sharing_level = SharingLevel.ORG
if shrEveryone:
    sharing_mgr.sharing_level = SharingLevel.EVERYONE
if shrGroups:
    for groupID in shrGroups:
        group = gis.groups.get(groupID)
        item_grp_sharing_mgr = sharing_mgr.groups
        item_grp_sharing_mgr.add(group=group)

I did notice if the sharing is already set on the feature service, you can omit this entirely and it will be maintained.  I can't recall if it use to do this or not at earlier versions of the API.

DJB
by
Regular Contributor

@JakeSkinner 

Thanks for the assistance Jake.  I wasn't aware that when overwriting a feature service it will still honour the original sharing properties.

I will definitely use this new code for when I need to alter sharing properties in the future.

Thanks again for your help Jake.  Cheers!

cjenkins_rva
Frequent Explorer

Hi @JakeSkinner, thanks very much for sharing this. Unfortunately, when I run the script, my symbology and pop-up configurations (both having been defined in the Visualization tab) are not preserved. I believe that's happening because that info is stored at the portal item level, not at the service level, so they aren't captured in existingDef (they would have be to grabbed via fsItem.get_data()). But in your video, I see these properties are preserved. I can't figure how that's possible. What am I missing?

KristalWalsh
Frequent Contributor

Hi @JakeSkinner thank you for this! I am not a programmer but I tried this and am getting what seems to be a sign-in to portal error. Do you know if anyone else has had this problem? I know I'm using the correct user name and password. I am using AGOL not Enterprise so this should be pretty straightforward I think. Thank you, Kristal

JakeSkinner
Esri Esteemed Contributor

@KristalWalsh do you know if you are using a built-in AGOL account?  Or, are you using a SAML user account?  With SAML, there is an icon you can click that will sign you in using the same account you typically sign into Windows with.

KristalWalsh
Frequent Contributor

@JakeSkinner hi, thank you, I have an organizational account for a government agency so not sure if that is considered "built-in". I am signed in to my AGO account in another window not that it makes any difference. 

JakeSkinner
Esri Esteemed Contributor

@KristalWalsh is there an '@' symbol in your username?

KristalWalsh
Frequent Contributor

@JakeSkinner no, not in the body of the code where my user name is inserted. I followed your video exactly.

KristalWalsh
Frequent Contributor

@JakeSkinner I noticed earlier that I was not signed in to Pro. I thought maybe I got disconnected at some point, so I just ran it again after signing in to Pro. It seems that when I execute the script, it signs me out of Pro. Should that happen?

JakeSkinner
Esri Esteemed Contributor

@cjenkins_rva I'm not sure if something changed in the API or not, but I swore this was working before.  However, I updated the code to use the .get_data() and apply this to the service using the requests module.  I couldn't find a way via the API to apply the properties all at once for all layers; that functionality may not exist, yet.  Anyways, give the updated code a try.

ModernElectric
Frequent Contributor

@JakeSkinner 

Been using the latest script to rebuild my update workflow.

On one of my Web Maps/Hosted Feature Layers - using this updated script - why am I getting:

mp = prj.listMaps(map)(0)

IndexError: list index out of range

 

JakeSkinner
Esri Esteemed Contributor

@ModernElectric do you have the name of the map in ArcGIS Pro specified correctly for the map variable?

JakeSkinner_0-1726665486634.png

 

ModernElectric
Frequent Contributor

Disregard.

Please forgive my ignorance, lack of proof-reading abilities 😉 

Updated Script is working perfectly for updating our AGOL dataset.

cog_GIS_Admin
Emerging Contributor

This script seems to be just what I am looking for, but I get an error when trying to login to AGOL.

Connecting to AGOL
Traceback (most recent call last):
File "E:\Scripts\Workorders_CD\NewOverwriteWebLayers.py", line 30, in <module>
arcpy.SignInToPortal(portal, user, password)
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\__init__.py", line 2609, in SignInToPortal
return _SignInToPortal(*args, **kwargs)
ValueError: Error signing on to https://arcgis.com/.

Message : s
Details : Unable to generate token.

 

Any Thoughts?

cog_GIS_Admin
Emerging Contributor

My post keeps getting marked as spam, not sure why. I'll try again. This script is just what I was looking for, but I am getting an error trying to log into AGOL.

 

Connecting to AGOL
Traceback (most recent call last):
File "E:\Scripts\Workorders_CD\NewOverwriteWebLayers.py", line 30, in <module>
arcpy.SignInToPortal(portal, user, password)
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\__init__.py", line 2609, in SignInToPortal
return _SignInToPortal(*args, **kwargs)
ValueError: Error signing on to https://arcgis.com/.

Message : s
Details : Unable to generate token.

 

Any ideas?

 

JakeSkinner
Esri Esteemed Contributor

@cog_GIS_Admin check the casing of your username, this is case sensitive.  For example, if your AGOL login is cog_GIS_Admin, and you specify cog_gis_admin, it will not authenticate.....even though cog_gis_admin will work in a web browser.

cog_GIS_Admin
Emerging Contributor

Thanks Jake, that was the issue. Now working on getting it to auto run with the server task scheduler.

JakeSkinner
Esri Esteemed Contributor

@cog_GIS_Admin here is helpful document on how use Windows Task Scheduler with python scripts:

https://community.esri.com/t5/python-documents/schedule-a-python-script-using-windows-task/ta-p/9158...

TomShewring
Emerging Contributor

I am running this script with an internal Enterprise (Portal) with IWA.

The script is working for me up until 'Update service with previous properties' 

Traceback (most recent call last):
File "<string>", line 81, in <module>
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

JakeSkinner
Esri Esteemed Contributor

@TomShewring are you attempting to update a hosted feature service, or a referenced service?

TomShewring
Emerging Contributor

Hi @JakeSkinner , a hosted feature service.
I have got the 'Update service with previous properties' working - I had to use the siteadmin (default Administrator) account when connecting to Enterprise (Portal).
I now have a error in the 'Clearing scratch directory' section. I do not have ArcGISPro open (I am running this script from the command line not from within ArcGISPro) - so I do not know what application could be locking the WebUpdate.sd?
Traceback (most recent call last):
File "E:\AGOL_Scripts\Geoprocessing\ScheduledTools\NHLEonPortal\updateNHLEonPortal2.py", line 102, in <module>
arcpy.Delete_management(file)
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\management.py", line 7665, in Delete
raise e
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\management.py", line 7662, in Delete
retval = convertArcObjectToPythonObject(gp.Delete_management(*gp_fixargs((in_data, data_type), True)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\geoprocessing\_base.py", line 512, in <lambda>
return lambda *args: val(*gp_fixargs(args, True))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
arcgisscripting.ExecuteError: ERROR 000601: Cannot delete C:\Users\tshewring\AppData\Local\Temp\scratch\WebUpdate.sd. May be locked by another application.
Failed to execute (Delete).

JakeSkinner
Esri Esteemed Contributor

@TomShewring I've seen this error occur sporadically in some implementations (mine included), and I'm unable to find a cause.  To workaround the issue, I recommend moving the Clear scratch directory section to the beginning of the script.  For example, move this section of code above the section that connects to AGOL/Portal.

TomShewring
Emerging Contributor

@JakeSkinner , thanks - moving the 'Clear scratch directory' section to the beginning of the script, above the section that connects to AGOL/Portal - this works.
One further thing I don't understand is - I have to use the <portalurl>:7443/arcgis address. If I use the WebAdaptor address I get errors such as these (even though I am using the same username and password in each scenario) -
Traceback (most recent call last):
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\auth\_auth\_winauth.py", line 75, in __init__
creds = gssapi.raw.acquire_cred_with_password(
File "gssapi\raw\ext_password.pyx", line 75, in gssapi.raw.ext_password.acquire_cred_with_password
gssapi.raw.exceptions.BadNameError: Major (131072): An invalid name was supplied, Minor (2529639136): Configuration file does not specify default realm

File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\auth\_auth\_winauth.py", line 84, in __init__
raise Exception("Please ensure gssapi is installed")
Exception: Please ensure gssapi is installed

Do you know why this is?

JakeSkinner
Esri Esteemed Contributor

@TomShewring 

  1.  Do you have Windows Authentication enabled on Portal's web adaptor?
  2. What version of Enterprise?
  3. What version of the arcgis api for python?

JakeSkinner_0-1732035419924.png

 

TomShewring
Emerging Contributor

@JakeSkinner 
1. Yes Windows Authentication is enabled on Portal WebAdaptor
2. Enterprise (Portal / Federated Server / Datastore) version 11.3
3. I have tried the script with

(i) ArcGISPro 3.3.1 - 

TomShewring_0-1732036450886.png

and

(ii) ArcGISPro 3.1.4

TomShewring_1-1732036599716.png

JakeSkinner
Esri Esteemed Contributor

@TomShewring try removing the username/password variables from the following lines:

JakeSkinner_0-1732037379063.png

These should not be needed if you Windows Authentication is enabled.  It will use the window's account that's signed into the server.

TomShewring
Emerging Contributor

Hi @JakeSkinner , changing to these settings (and using the Portal WebAdaptor address) -
when I run the script from the command line I get a security challenge

TomShewring_0-1732090749644.png

So I could not run this as a scheduled task on the Windows server

JakeSkinner
Esri Esteemed Contributor

@TomShewring this should not occur if you're running command prompt as a named user in Portal, and you have Portal's URL added to the Local Intranet (i.e. Start > Internet Options > Security tab > Local Intranet > Sites > Advanced > specify the portal URL (minus the web adaptor) > Add):

JakeSkinner_0-1732107076433.png

 

TomShewring
Emerging Contributor

@JakeSkinner thats sorted it. Thanks.
Although we had deployed the Local Intranet change to all users, it had not been deployed on the server running the Overwrite (Portal) Feature Service script. Thanks for your help. Much appreciated.

Tharu20B
Occasional Contributor

Wow!!

I learned lot of things from this. Thank you. 

haksinguiliam
New Contributor

This is fantastic! 🙏

One suggestion to the recording, which by the way, did an excellent job of listing possible changes user like me would potentially make, is having a couple view layer created. I would assume using the script wouldn't disconnect view layer or receive any failed update error, but it would be comforting to see in the video. 

 

SenadHodzicNR
Emerging Contributor

@JakeSkinner  So reading through the script, no matter what you still need to disable sync to update the  feature service? Has anyone tested to see if their Offline Map Areas are preserved after this script is run? 

Version history
Last update:
‎12-07-2024 11:55 AM
Updated by:
Contributors