Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

[MikeVolz]

I am trying to append features to an AGOL HFL from a Geojson file with ArcGIS API for Python, but it is very slow.  I do not know if there is some sort of size limit for this procedure, as I am attempting to append 14K point features with about 20 columns to the AGOL HFL.

Has anyone attempted such an operation with success?

Any help or hints are greatly appreciated.  Thanks in advance.

[AustinAverill]

I have a much, much larger point file that I append as part of some automations I have set up. What does your script look like and what amount of time are you calling long?

[MikeVolz · Accepted solution]

Below is the script I am trying to use.  The script has been running for several hours now on a Windows Server server where Task Manager Python CPU is at 0%, but IDLE has not indicated that the script has finished and the script has not thrown any errors.

import arcgis
import arcpy
from arcgis import GIS

user = "User Name"
password = "Password"
item_id = "AGOL Item ID"

gis = GIS(r"AGOL URL",user, password)

data_id = gis.content.get(item_id)

print(data_id)
layer = FeatureLayer.fromitem(item=data_id, layer_id=0)

print(layer)
print(" Truncate features started")
layer.manager.truncate()
print(" Truncate features completed successfully!!!\n")

# Get file path to upload geojson file
file_path = "File path and file name"

geojson = gis.content.add({
"type": "GeoJson",
"title": "Name of uploaded Geojson file in AGOL",
"tags": ["Tags"]
}, data=file_path)
geojson

print(" Geojson item id = " + geojson.id)

FeatLay = data_id.layers[0]
FeatLayUrl = FeatLay.url
print(FeatLayUrl)
print(" Append features started")

status = data_id.layers[0].append(item_id = geojson.id, upload_format = 'geojson', source_table_name = "Common_Place_Geojson")
print(" Append features completed successfully!!!")

# Delete geojson file for next automated update
item_Delete = gis.content.get(geojson.id)
delete_result = item_Delete.delete()

[AustinAverill]

Gotcha, I wouldn't expect this script to take anymore than 5-10 minutes (depending on network performance on both yours and ESRI's side. Probably less.

Have you observed any of the print statements? If not, I would probably terminate the script manually and add in some logging. You could potentially be stuck trying to load imports or something else. Are you using an ArcGIS Pro created environment or have you made a specific venv?

[JakeSkinner]

Hi @MikeVolz, I see that you have the ability to run arcpy.  I find arcpy's Append tool more robust.  Instead of adding your GeoJSON file to Portal/AGOL, you could convert it to features using JSON to Features and then Append this feature class to your feature service.

[MichaelVolz]

Jake:

This was the original route I was taking, but I was trying to use the Truncate tool which is only available through ArcGIS API for Python so I had to log in twice, once for arcpy and then again for ArcGIS API for Python.

Can I use Delete Features tool to remove records from AGOL HFL before running the append tool?  If so, is that tool fast to process?

[MikeVolz]

Jake:

The script I attached did actually work, just not on its initial run.  I also noticed that after the script ran successfully the first time some of the fields in the HFL that was truncated did not get all fields correctly populated with the Append process.  I ran the script again and the fields were then populated so that was a false negative which I initially thought might be a bug.

I then added this script to a Scheduled Task and confirmed that it is a repeatable successful process where no field values are being lost in the Append process.

Also note that this was a small point data set, so not sure the same results will occur on a more complex polygon data set with a much larger number of features.

[AustinAverill]

Mike, this script should work well. The only issues that you may run into with field matching like described (that I can think of) is how the geojson is being generated. If you are passing this through a shpefile format at some point its possible that fieldnames are being altered, or if your field names don't match the hosted service to start with (spaces in column names, etc.) The Append function does a pretty good job of trying to recognize small discrepancies and reconcile them, but is far from a miracle worker! 

Might be useful to save and examine the GEOJSON file on the next couple updates and make sure that the source file you are appending is matching 1:1 with the given dataset schema.

[MikeVolz]

Austin:

Based on previous experience with shapefiles truncating field names that were created on purpose with more than 10 characters for clarity, I avoid using shapefiles in processes such as this.

Also what software do you use to view a GEOJSON file to make it more human readable?  Looking at GEOJSON in Google Chrome or Microsoft Edge is difficult.

[AustinAverill]

You can import it into Pro or Google Earth and see the field names and all pretty easily. Or you can just read the GEOJSON on using geopandas and examine the columns from the geodataframe.