Unable to update point feature layer in AGOL using Python

[HaroldDoran2]

I am having trouble updating a point feature layer using the most recent API 2.4.1. I have read more posts and blogs and still have not had any luck making this work. What I am trying to do is update a point layer in AGOL using python. This layer is then consumed by other web maps and eventually a dashboard. I've tried the truncate/append process only to successfully truncate the data but have not found the magic line of code that will make the append work. The workflow involves using a CSV file that is updated daily and then pushing that up to AGOL. 

Can anyone shed some light as to what I am doing wrong? I cannot find any detailed information on the new object model for the API that I'd usually use as a roadmap to solve this but the current documentation is quite lacking in examples/details.

Any insights/help would be greatly appreciated. Thanks in advance.

Here is my code:

import agol_utils
from arcgis.layers import Service
import arcpy
from arcpy import env
from arcpy.sa import *
import arcgis
from datetime import datetime as dt
import os, zipfile, glob
import sys
import json
from arcgis.features import FeatureLayer

def TruncateWebLayer(gis=None, target=None):
try:
lyr = arcgis.features.FeatureLayer(target, gis)
lyr.delete_features(where="1=1")
print ("Successfully truncated layer: " + str(target))
except:
print("Failed truncating: " + str(target))
sys.exit()

def main(location, folder_name, data_source, upload_data_location, raster_names, csv_file_name):
try:
#get the current date for file naming etc.
current_dt = str(int(dt.now().timestamp()))
#connect with AGOL
my_gis = agol_utils.connect_to_agol(location)

#make sure I'm logged in and know which version of API I am using
username = my_gis.users.me.username
print(username)
print(arcgis.__version__)

#get the folder the user wants to put the results in on AGOL
content_manager = arcgis.gis.ContentManager(my_gis)
folders_obj = my_gis.content.folders #
item_folder = folders_obj.get(folder=folder_name)
print(f"Your folder is: {item_folder}")

#AFFES anomalies results point layer item id
csv_item_id = "5481ace750b0474d804740e28d2d43a2"
csv_existing_item = my_gis.content.get(csv_item_id)

csv_feature_layer_url = "https://services1.arcgis.com/TJH5KDher0W13Kgo/arcgis/rest/services/AFFES_Anomaly_Results/FeatureServ..."

csv_feature_layer = FeatureLayer(csv_feature_layer_url)
update_features = r"C:\TEMP\TESTING\INPUT\aware_testing.gdb\percentile_CoordinateTableToPoint"
localJSON = r"C:\TEMP\TESTING\INPUT\csvjson.json"

#remove features
#TruncateWebLayer(my_gis,csv_feature_layer)
csv_feature_layer.delete_features(where="1=1")

#reference the empty layer as FeatureLayer object from the ArcGIS Python API
fl = arcgis.features.FeatureLayer(csv_feature_layer, my_gis)

#create a JSON object from the local features
jSON = arcpy.FeaturesToJSON_conversion(update_features, localJSON)

#create a FeatureSet object from the JSON
fs = arcgis.features.FeatureSet.from_json(open(localJSON).read())

#add/append the local features to the hosted feature layer.
fl.edit_features(adds = fs)

print("Feature layer updated")

except arcpy.ExecuteError:
print(arcpy.GetMessages(2))
except Exception as e:
print(f"An error occurred: {e}")

if __name__ == "__main__":

csv_update_file_name = r'percentile.csv'
input_file_path = os.path.join(r"C:\TEMP\TESTING\INPUT",csv_update_file_name)
print(input_file_path)

main("home","Situational_Awareness",r"c:\temp\input",r"c:\temp\output","bui_interpolation, dc_interpolation, dmc_interpolation", input_file_path)

[Clubdebambos · Accepted solution]

Hi @HaroldDoran2,

Plenty of ways to do this as you have probably come across on your research, many hit and miss and dependent of workflows. Here is one that I use from time to time.

Basic overview of workflow...
- Truncate the FeatureLayer (won't work if sync is set or the layer is an origin in a relationship - you can delete_features instead)
- Add the CSV as an Item to AGOL
- Publish CSV as a Service (I do as a table to save on credits)
- Append tabular data from CSV to FeatureLayer without Geometry
- Get the Point geometry information from the attributes and apply

Note: for this workflow the field names in the FeatureLayer always match the field names in the CSV.

Perhaps test on a disposable FeatureLayer and see if it works for your needs?

from arcgis.gis import GIS, ItemProperties, ItemTypeEnum
from arcgis.features import FeatureLayer, FeatureSet

################################################################################
## ACCESS ARCGIS ONLINE ########################################################

##agol = GIS("home")
agol = GIS("home")

################################################################################
## REQUIRED OBJECTS ############################################################

## get the Feature Layer as a FeatureLayer object
fl_url = "FEATURE_LAYER_URL" # insert the url to the feature layer here

## path to csv file
csv_file = r"C:\path\to\update_agol.csv"

## home folder to add CSV item an table service, these will be deleted
folder = agol.content.folders.get()

## Item properties required when using folder.add()
csv_item_properties = ItemProperties(
    title = "UPLOADED_CSV",
    item_type = ItemTypeEnum.CSV.value
)

## the fields that contain the X and Y values
x_field = "LONGITUDE"
y_field = "LATITUDE"

################################################################################
## TRUNCATE FEATURE LAYER ######################################################

fl = FeatureLayer(
    url = fl_url
)

fl.manager.truncate() # must not be in a relationship or have sync enabled

## OR

## fl.delete_features(where="1=1")

################################################################################
## CSV DATA TO FEATURELAYER: TABLE ONLY - WITHOUT GEOMETRY #####################


## add as csv item in AGOL
csv_agol = folder.add(
    item_properties=csv_item_properties,
    file=csv_file
).result()

## publishing properties so csv is added as a table and not a spatial layer.
## you could publish as a spatial layer and consume credits and then you wouldnt
## have to accomodate for geomtry later on
csv_properties = {
    "type:" : "csv",
    "locationType" : "none"
}

## publish hosted table
csv_ht = csv_agol.publish(publish_parameters=csv_properties)

## use analyze as per append docs when using a csv
csv_data = agol.content.analyze(
    item = csv_agol.id,
    file_type = "csv",
    geocoding_service = None,
    location_type = None
)

## append in data
fl.append(
    item_id = csv_agol.id,
    upload_format = "csv",
    source_info = csv_data,
    upsert = False
)

################################################################################
## UPDATE GEOMETRY #############################################################

## re-get the FeatureLayer object
fl = FeatureLayer(
    url = fl_url
)

## get the current features
fs = fl.query()

## convert FeatureSet object to a dictionary
fs = fs.to_dict()

## add geometry to each feature
for f in fs["features"]:
    f["geometry"] = {"x" : f["attributes"][x_field], "y" : f["attributes"][y_field], "spatialReference" : {"wkid" : 4326, "latestWkid" : 4326}}

## convert dictionary back to a FeatureSet object
fs = FeatureSet.from_dict(fs)

## update the records with the geomtry
fl.edit_features(updates=fs)

################################################################################
## DELETE CSV TEM AND SERVICE ITEM #############################################

csv_agol.delete(permanent=True)
csv_ht.delete(permanent=True)

################################################################################

All the best,

Glen

[HaroldDoran2]

Thanks Glen I will give that a go and if it works I'll mark it as solved.

cheers

Harold

[HaroldDoran2]

Hi Glen,

It works...thank you very much for the insight and code. 

How would you expand this to a polygon or line feature class? Thoughts?

thanks

Harold