Automate overwrite web layer, feature class

37080
79
06-14-2018 12:53 AM
EinarDørheim1
New Contributor III

Hi 

I'm trying to automate an update process for one of my hosted features in portal. So far I have:

1. Solved the automatic download and adjustment of the file in question. (using powershell/task scheduler)

2. exportet the python code that runs the "Geocode Addresses" tool on my file, saves it in a file gdb. 

3. manually published the feature class with the "overwrite web layer" function. 

My challange is to automate step 3, and connect it to step 2. 

Please keep in mind that I'm completely new to scripting/python. 

I'm currently using ArcGIS Pro 2.1.0, and have admin rights on the portal. 

I have looked at thise sites, and think they describe partly how to solve my issue, I'm just not able to pick out the relevant parts and build a script that works for my particular problem..

community.esri.com "using python to overwrite a feature layer"

developers.arcgis.com python, overwriting feature layers

esri updating-your-hosted-feature-services-with-arcgis-pro-and-the-arcgis-api-for-python

79 Replies
RehanChaudhary
Occasional Contributor

Hi @JakeSkinner is there any such workaround for a raster dataset? i can publish it by creating SD files but i am struggling to update it.? Also if there is a way to preserve the symbology

0 Kudos
JaredPilbeam2
MVP Regular Contributor

It doesn't look like anyone has mentioned this help page: https://pro.arcgis.com/en/pro-app/latest/arcpy/sharing/featuresharingdraft-class.htm

  • The FeatureSharingDraft class allows you to create a service definition draft file (.sddraft) for a web feature layer  that copies all data to either ArcGIS Enterprise or ArcGIS Online.

I needed to overwrite an existing feature service in AGOL. I adapted one of the code examples from the help page. I'm referencing a map layer within a Pro project from a stand-alone script.

 

import arcpy
import os

'''
Publish a layer
The following script publishes a layer in a map as a web feature layer to a portal.
'''

# Sign in to portal
arcpy.SignInToPortal("https://gis.maps.arcgis.com/", "user", "pass")

arcpy.env.overwriteOutput = True

# Set output file names
outdir = r"C:/Users/folder"
service_name = "TEST"
sddraft_filename = service_name + ".sddraft"
sddraft_output_filename = os.path.join(outdir, sddraft_filename)
sd_filename = service_name + ".sd"
sd_output_filename = os.path.join(outdir, sd_filename)

# Reference layers to publish
aprx = arcpy.mp.ArcGISProject(r"\pathto\Project.aprx")
m = aprx.listMaps('Tot Pop')[0]
selected_layer = m.listLayers()[0]
print(selected_layer)
# selected_table = m.listTables()[0]

# Create FeatureSharingDraft and set overwrite property
server_type = "HOSTING_SERVER"
sddraft = m.getWebLayerSharingDraft(server_type, "FEATURE", service_name, [selected_layer])
sddraft.overwriteExistingService = True

# Create Service Definition Draft file
sddraft.exportToSDDraft(sddraft_output_filename)

# Stage Service
print("Start Staging")
arcpy.StageService_server(sddraft_output_filename, sd_output_filename)

# Share to portal
print("Start Uploading")
arcpy.server.UploadServiceDefinition(sd_output_filename, server_type)

print("Finish Publishing")

 

 

by Anonymous User
Not applicable

Just be aware this method or most of the other methods don't work if you've got outstanding replicas or sync enabled. 

AdminAccount2
Occasional Contributor II

Would be great if someone could modify this script to include disable sync and unregister replicas before publishing takes place. I actually have attached a script that does this (attached) ... but not sure how/where to add it into the existing code example for automated publishing.

 

from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection
from arcgis.features import FeatureLayer
import requests

# Connect to your ArcGIS Online organization
gis = GIS("https://www.arcgis.com","username","password")
token = gis._con.token

#Hosted feature layer names to disable sync and unregister all replicas
search_queries = ["Test Layer1", "Test Layer2"] 

# Create an empty list to store the item IDs
item_ids = []

# Loop through each search query and get the corresponding feature services
for search_query in search_queries:
    search_results = gis.content.search(query=search_query, item_type="Feature Service")
    #print (search_results)
    

    # Loop through each search result and append the item ID to the list
    for search_result in search_results:
        item_ids.append(search_result.id)

#print(item_ids)

# Loop through each item ID and unregister all replicas then disable sync
for item in item_ids:
    flc = FeatureLayerCollection.fromitem(gis.content.get(item))
    url = flc.url + "/unRegisterReplica?&replicaID=*&f=json&token="+token
    #print (url)
    response = requests.post(url)
    flc.manager.update_definition({"syncEnabled": False})

message = "Successfully unregistered all replicas and disabled sync on: " + ", ".join(search_queries)
print(message)

 

 

 

0 Kudos
JakeSkinner
Esri Esteemed Contributor

@AdminAccount2 take a look at the following document.  The script has an option to unregister existing replicas.

AdminAccount2
Occasional Contributor II

HI Jake - thanks so much that script works great! Pretty much does what I want to do, but I do have some suggestions that I think would make it even better if possible!

Ideally this script would loop through a list of all the layers the map and republish each one of them. The values for the variables named serviceDefID and featureServiceID could be derived automatically from the existing hosted services by matching up the layer names in the Pro map with existing published service names (assuming that the map layer names must match the published feature service names for this to work). 

Since the real value here is to automate publishing, editing the script each time to change those variables and run it one at a time for each layer is probably going to take more time than just overwriting them manually. Of course you could create multiple scripts with different variables for each layer then run them all, but that's not ideal either. In a perfect world, you would have a Pro project with map that included all the layers to be overwritten and the script would process each of those layers without any intervention. 

The other thing is I would only update the Sharing properties if the value of  shrGroups is not null. If it is null, then just leave the existing sharing properties intact.

0 Kudos
JakeSkinner
Esri Esteemed Contributor

Hi @AdminAccount2,

Ideally this script would loop through a list of all the layers the map and republish each one of them.

A single service can have multiple layers, so this would not work in that scenario.  Also, different services may require different variables (i.e. sharing).  An option could be to create a config file of the variables that the code loops through and executes the overwrite.

0 Kudos
AdminAccount2
Occasional Contributor II

A single service can have multiple layers, so this would not work in that scenario.

True - but for the sake of reaping the benefits of using an automated publishing process like this, I'd make sure each layer in the map was published separately as a single service. This would be a special map only used for this purpose. The script will be run daily from a CI tool like Jenkins, so it needs to run completely autonomously on multiple layers.

AdminAccount2
Occasional Contributor II

I fixed up this script so it can be run autonomously from a CI tool. The script will connect to a Pro Project and look for a map that contains the feature layers to be overwritten. All the feature layers contained in this map and their service definitions must have already been published initially and hosted on AGOL.

The script disables sync and unregisters replicas before overwriting the service definition file and existing feature service. The feature layers in the map may be grouped or ungrouped. Grouped layers will be published as a single feature service, same as the ungrouped layers. All service properties (sync, sharing, editing etc.) will remain the same as they originally were. The layer names in the Pro map must match the published feature layer names, so make sure the map is setup correctly in Pro. 

Thanks to Jake Skinner for getting this started in the right direction.

 

import arcpy, os, time
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection

# Variables
prjPath = r"C:\PathToProProject\ProProject.aprx"    # Path to Pro Project
map_name = 'Test Map Overwrite'                     # Name of map in Pro Project
portal = "https://www.arcgis.com"                   # AGOL
user = "username"                                   # AGOL username
password = "password"                               # AGOL password
unregisterReplicas = True                           # True/False to unregister existing replicas
group_layer_names = []
ungrouped_layer_names = []

# Set Environment Variables
arcpy.env.overwriteOutput = 1

# Start Timer
startTime = time.time()

print(f"Connecting to AGOL")
gis = GIS(portal, user, password)

project = arcpy.mp.ArcGISProject(prjPath)

# Get the map in the project
map_obj = project.listMaps(map_name)[0]

# Iterate through all layers in the map and list the grouped layer names
for layer in map_obj.listLayers():
    if layer.isGroupLayer:
        group_layer_names.append(layer.name)

print (f"Grouped layer names: {group_layer_names}")

# Iterate through all layers in the map and list the ungrouped layer names
for layer in map_obj.listLayers():
    if not layer.isGroupLayer:
        # Check if the layer is part of any group layer
        is_in_group = False
        for parent_layer in map_obj.listLayers():
            if parent_layer.isGroupLayer and layer in parent_layer.listLayers():
                is_in_group = True
                break
        if not is_in_group:
            ungrouped_layer_names.append(layer.name)

print (f"Ungrouped layer names: {ungrouped_layer_names}")

layer_list = ungrouped_layer_names + group_layer_names
print (f"Layers to be republished: {layer_list}")

# Loop through all the layers and get the feature layer and service definition ID's
for layer_name in layer_list:

    # Filter the feature layers for an exact match and get the feature layer ID
    fl_search_results = gis.content.search(query=f"title:{layer_name}", item_type="Feature Layer", sort_field='title', sort_order='asc')
    fl_item = None
    # Iterate through the search results to find the exact match
    for item in fl_search_results:
        if item.title == layer_name:
            fl_item = item
            break

    if fl_item:
        print(f"Found exact match for feature layer: {fl_item.title} (ID: {fl_item.id})")
    else:
        print(f"No exact match found for feature layer: {layer_name}")

    # Filter the service definitions for an exact match and get the service definition ID
    sd_search_results = gis.content.search(query=f"title:{layer_name}", item_type="Service Definition", sort_field='title', sort_order='asc')
    sd_item = None
    # Iterate through the search results to find the exact match
    for item in sd_search_results:
        if item.title == layer_name:
            sd_item = item
            break

    if sd_item:
        print(f"Found exact match for service definition: {sd_item.title} (ID: {sd_item.id})")
    else:
        print(f"No exact match found for service definition: {layer_name}")

    # Local paths to create temporary content
    sddraft = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sddraft")
    sd = os.path.join(arcpy.env.scratchFolder, "WebUpdate.sd")
    sdItem = gis.content.get(sd_item.id)
    #print (sdItem)

    # Create a new SDDraft and stage to SD
    print("Creating SD file")
    arcpy.env.overwriteOutput = True
    prj = arcpy.mp.ArcGISProject(prjPath)
    mp = prj.listMaps(map_name)[0]
    serviceDefName = sdItem.title
    arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, serviceDefName, 'MY_HOSTED_SERVICES', 'FEATURE_ACCESS', '', True, True)
    arcpy.StageService_server(sddraft, sd)

    # Reference existing feature service to get properties
    fsItem = gis.content.get(fl_item.id)
    #print (fsItem)
    flyrCollection = FeatureLayerCollection.fromitem(fsItem)
    existingDef = flyrCollection.properties

    # Unregister existing replicas
    if unregisterReplicas:
        if flyrCollection.properties.syncEnabled:
            print("Unregister existing replicas")
            for replica in flyrCollection.replicas.get_list():
                replicaID = replica['replicaID']
                flyrCollection.replicas.unregister(replicaID)

    # Overwrite feature service
    sdItem.update(data=sd)
    print("Overwriting existing feature service")
    fs = sdItem.publish(overwrite=True)

    # Update service with previous properties
    print("Updating service properties")
    flyrCollection.manager.update_definition(existingDef)

    print("Clearing scratch directory")
    arcpy.env.workspace = arcpy.env.scratchFolder
    for file in arcpy.ListFiles():
        if file.split(".")[-1] in ('sd', 'sddraft', 'png', 'xml'):
            arcpy.Delete_management(file)

    endTime = time.time()
    elapsedTime = round((endTime - startTime) / 60, 2)
    print(f"Script completed in {elapsedTime} minutes")

 

 

 

0 Kudos
GrimsbyGIS
New Contributor

Hi AdminAccount2, thanks so much for sharing this script, i was able to use it solve an issue i had where the project contains multiple layers that require updating. Do you know how this script could be modified so that the layer scheme's could be updated as well, or is this not supported using the overwrite function?

0 Kudos