Automate overwrite web layer, feature class

[EinarDørheim1]

Hi 

I'm trying to automate an update process for one of my hosted features in portal. So far I have:

1. Solved the automatic download and adjustment of the file in question. (using powershell/task scheduler)

2. exportet the python code that runs the "Geocode Addresses" tool on my file, saves it in a file gdb. 

3. manually published the feature class with the "overwrite web layer" function. 

My challange is to automate step 3, and connect it to step 2. 

Please keep in mind that I'm completely new to scripting/python. 

I'm currently using ArcGIS Pro 2.1.0, and have admin rights on the portal. 

I have looked at thise sites, and think they describe partly how to solve my issue, I'm just not able to pick out the relevant parts and build a script that works for my particular problem..

community.esri.com "using python to overwrite a feature layer"

developers.arcgis.com python, overwriting feature layers

esri updating-your-hosted-feature-services-with-arcgis-pro-and-the-arcgis-api-for-python

[KevinBelton]

I'm trying to run this script, it works fine from my laptop outwith our corp network but when I try to run the python script either within ArcPro or as a standalone python script it fails as it cannot connect to the port url to login. It times out and throws a few errors which I can only presume is due to our proxy. I can login into the port fine via ArcPro and overwrite the layer via ArcPro tools I just can't run the python script.

urllib.error.URLError: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>

[RoryBennison]

Thank you to those who have contributed to this discussion, I have now found a solution that allows us to successfully update our hosted feature layers, I found a couple of fish hooks with searcing for the feature layer so have modified the script to use the item id, here is the version we are running.  I am very new to scripting so my future modification would be to include a loop that runs through all projects within a folder and a list of item id's.  But that is for the future.

import arcpy
import os, sys, time
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection

startTime = time.clock()

#Sign In Details
portal = "https://www.arcgis.com/"
user = "abc"
password = "123"

# Start setting variables

#itemid of feature layer to set sync on
itemid = "d14a769e29a14d62ad6d311368544290"

# Set the path to the project
prjPath = r"L:\ArcGIS Pro Projects\Daily Updates\Harvest Settings\Harvest Settings.aprx"

# Update the following variables to match
sd_fs_name = "Harvest Settings"
sditemid = "fafc376ae784479d9d5aa338201de994"

# Set sharing options
shrOrg = ""
shrEveryone = ""
shrGroups = ""

# End setting variables

# Local paths to create temporary content
relPath = r"L:\scratch\TempSD"
sddraft = os.path.join(relPath, "WebUpdate.sddraft")
sd = os.path.join(relPath, "WebUpdate.sd")

#Create a new sddraft and stage to sd
print("Creating SD file...")
arcpy.env.overwriteOutput = True
prj = arcpy.mp.ArcGISProject(prjPath)
mp = prj.listMaps()[0]

arcpy.mp.CreateWebLayerSDDraft(map_or_layers=mp,
out_sddraft=sddraft,
service_name=sd_fs_name,
server_type="HOSTING_SERVER",
service_type="FEATURE_ACCESS",
folder_name=None,
overwrite_existing_service=True,
copy_data_to_server=True,
enable_editing=True,
allow_exporting=True,
enable_sync=True)

arcpy.StageService_server(sddraft, sd)

print("Connecting to {}".format(portal))
gis = GIS(portal, user, password)

# Find the sd, update it, publish it with overwrite and set sharing and metadata
print("Search for original SD on portal...")
sdItem = gis.content.get(sditemid)
print("Found SD: {}, ID: {} n Uploading and Overwriting...".format(sdItem.title, sdItem.id))
sdItem.update(data=sd)
print("Overwriting existing feature service...")
fs = sdItem.publish(overwrite=True)
print("Finished updating: {} - ID: {}".format(fs.title, fs.id))
#Set Sync on hosted feature layer
print ("Setting Sync Layer")
fc=gis.content.get(itemid)
fc= FeatureLayerCollection.fromitem(fc)
update_dict2 = {"capabilities": "Query",
"syncEnabled": True}
fc.manager.update_definition(update_dict2)
fc.properties.capabilities

endTime = time.clock()
elapsedTime = round((endTime - startTime) / 60, 2)
print("Script finished in {0} minutes".format(elapsedTime))

[ReggieSanders]

Code was working great but all of a sudden once it creates the service definition it looks nothing like the way I have it in the aprx. Its like its defaulting to another one some where any ideas?

[PatrickMcKinney1]

I was happy to find this thread!  I also found using the suggestion of gis.content.get(sd_id) allowed me to automate overwriting my feature service.

My project is for a countywide tax parcels layer.  Whenever I update the layer (ovewrite feature service) I also like to "Optimize Layer Drawing" and "Rebuild Indexes."  

Are there some Python API commands to automate these tasks as well?

[RoryBennison]

I haven’t explored those functions, my suggestion would be to have a look at the documentation for the arcgis api, there is possibly something in there. I followed the example here https://developers.arcgis.com/python/guide/updating-feature-layer-properties/

Cheers

[Anonymous User]

Hello,

Any idea how to specify a time zone when overwriting the layer? I was able to get this code to work but my data is showing up one day behind because it's not converting from UTC to local time.

Thanks!

[Anonymous User]

See this thread for specifying a time zone CreateWebLayerSDDraft doesn't include Date Field configuration? 

[AmandaBishop2018]

Hello All,

I have been running the original script provided by Jake Skinner and it has been working.  But, with upgrades to desktop and ArcGIS Pro, I have run into some snags and I'm revisiting this script for automated updates to two of my feature services.  Below is the script that I am running and the error that I am receiving.  I have tried completely deleting the old feature service and service definition file from AGOL and starting over and I still get the error. If anybody would be so kind to assist, I would greatly appreciate it!

HERE IS THE SCRIPT:

import arcpy
import os, sys
from arcgis.gis import GIS

### Start setting variables
# Set the path to the project
prjPath = r"C:\Spam\PY\FeatureServiceUpdate\TaxParcel\TaxParcelsUpdate.aprx"

# Update the following variables to match:
# Feature service/SD name in arcgis.com, user/password of the owner account
sd_fs_name = "Tax_Parcels"
portal = "https://acpa.maps.arcgis.com" # Can also reference a local portal
user = "ACPA"
password = "***************"

# Set sharing options
shrOrg = True
shrEveryone = True
shrGroups = ""

### End setting variables

# Local paths to create temporary content
relPath = os.path.dirname(prjPath)
sddraft = os.path.join(relPath, "WebUpdate.sddraft")
sd = os.path.join(relPath, "WebUpdate.sd")

# Create a new SDDraft and stage to SD
print("Creating SD file")
arcpy.env.overwriteOutput = True
prj = arcpy.mp.ArcGISProject(prjPath)
mp = prj.listMaps()[0]
arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, sd_fs_name, "MY_HOSTED_SERVICES", "FEATURE_ACCESS","", True, True, allow_exporting=True)
arcpy.StageService_server(sddraft, sd)

print("Connecting to {}".format(portal))
gis = GIS(portal, user, password)

# Find the SD, update it, publish /w overwrite and set sharing and metadata
print("Search for original SD on portal…")
sdItem = gis.content.search("{} AND owner:{}".format(sd_fs_name, user), item_type="Service Definition")[0]
print("Found SD: {}, ID: {} n Uploading and overwriting…".format(sdItem.title, sdItem.id))
sdItem.update(data=sd)
print("Overwriting existing feature service…")
fs = sdItem.publish(overwrite=True)

if shrOrg or shrEveryone or shrGroups:
print("Setting sharing options…")
fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)

print("Finished updating: {} – ID: {}".format(fs.title, fs.id))

THIS IS HOW FAR I GET WITH THE SCRIPT AND THE ERROR I RECEIVE:

Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 14:00:49) [MSC v.1915 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>>
RESTART: C:\Spam\PY\FeatureServiceUpdate\TaxParcel\TaxParcelFeatureServiceUpdate.py
Creating SD file
Connecting to https://acpa.maps.arcgis.com
Search for original SD on portal…
Found SD: Public_Parcels, ID: cea64d901d7c4282879e600d9f34113d n Uploading and overwriting…
Overwriting existing feature service…
User cant overwrite this service, using this data, as this data is already referring to another service.
Traceback (most recent call last):
File "C:\Spam\PY\FeatureServiceUpdate\TaxParcel\TaxParcelFeatureServiceUpdate.py", line 45, in <module>
fs = sdItem.publish(overwrite=True)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 8818, in publish
folder, buildInitialCache)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\portalpy.py", line 372, in publish_item
resp = self.con.post(path, postdata, files)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 1183, in post
self._handle_json_error(resp_json['error'], errorcode)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py", line 1204, in _handle_json_error
raise RuntimeError(errormessage)
RuntimeError: User cant overwrite this service, using this data, as this data is already referring to another service.
(Error Code: 500)
>>>

[NicolasMarecos]

Hi Amanda, 

I found this post that refers to the problem you're running into.

Could you try to change:

sdItem = gis.content.search("{} AND owner:{}".format(sd_fs_name, user), item_type="Service Definition")[0]

To this:

sdItem = gis.content.search(query="title:"+ sd_fs_name + " AND owner: " + user, item_type="Service Definition")[0]

From what I read the original search string is causing the turn into a boolean, which does not restrict the title to an exact match. This may not be returning the desired item that needs to be overwritten.

[JaneSkillman]

Hello Everyone,

So, I'm a bit confused about why the script here involves updating the service definition and using an ArcPro project map, but the script here does not. Is there guidance concerning when to use one method vs. the other? 

I tried the linked sample script as a method to overwrite a hosted feature layer I published from ArcPro with a feature class in a file geodatabase. I don't get any errors, but it also doesn't update and I can't find any information on this. I would prefer not to have to maintain an ArcPro Project / map because I'm not the person who'll actually be running the script. Any suggestions appreciated. 

# Import libraries
from arcgis.gis import GIS
from arcgis import features
from arcgis.features import FeatureLayerCollection
import arcpy

# sign in to portal through ArcPro
gis = GIS("pro")

# new data
data_path =r"C:\path\to\gdb"
gdb = 'relevant.gdb'

arcpy.env.workspace = os.path.join(data_path, gdb)

fcs = arcpy.ListFeatureClasses()
fc = fcs[0]
print(fc)
#fc = "SouthAtlantic"

arcpy.GetCount_management(fcs[0])
# 6955 records at last count.

""" published feature layer 
I published this as a hosted feature class; 
the feature class name in the gdb 
matches the feature layer name in the feature layer collection. (SouthAtlantic)
The name of the gdb also matches the name of the gdb I originally used to upload the feature service in ArcPro, although I don't know if that's important.
"""
# get item from arconline
item = gis.content.get(itemID)

# overwrite feature layer collection with gdb
item_flayer_collection = FeatureLayerCollection.fromitem(item)
item_flayer_collection.manager.overwrite(os.path.join(data_path,gdb))

#{'success': True}

item.layers[0].query(return_count_only=True)

# 6908 records, which means it wasn't actually updated.