topic Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow in ArcGIS API for Python Questions

Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

MikeVolz — Tue, 15 Apr 2025 17:59:34 GMT

I am trying to append features to an AGOL HFL from a Geojson file with ArcGIS API for Python, but it is very slow. I do not know if there is some sort of size limit for this procedure, as I am attempting to append 14K point features with about 20 columns to the AGOL HFL.

Has anyone attempted such an operation with success?

Any help or hints are greatly appreciated. Thanks in advance.

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

AustinAverill — Tue, 15 Apr 2025 18:05:17 GMT

I have a much, much larger point file that I append as part of some automations I have set up. What does your script look like and what amount of time are you calling long?

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

MikeVolz — Tue, 15 Apr 2025 20:37:19 GMT

Below is the script I am trying to use. The script has been running for several hours now on a Windows Server server where Task Manager Python CPU is at 0%, but IDLE has not indicated that the script has finished and the script has not thrown any errors.

import arcgis
import arcpy
from arcgis import GIS

user = "User Name"
password = "Password"
item_id = "AGOL Item ID"

gis = GIS(r"AGOL URL",user, password)

data_id = gis.content.get(item_id)

print(data_id)
layer = FeatureLayer.fromitem(item=data_id, layer_id=0)

print(layer)
print(" Truncate features started")
layer.manager.truncate()
print(" Truncate features completed successfully!!!\n")

# Get file path to upload geojson file
file_path = "File path and file name"

geojson = gis.content.add({
"type": "GeoJson",
"title": "Name of uploaded Geojson file in AGOL",
"tags": ["Tags"]
}, data=file_path)
geojson

print(" Geojson item id = " + geojson.id)

FeatLay = data_id.layers[0]
FeatLayUrl = FeatLay.url
print(FeatLayUrl)
print(" Append features started")

status = data_id.layers[0].append(item_id = geojson.id, upload_format = 'geojson', source_table_name = "Common_Place_Geojson")
print(" Append features completed successfully!!!")

# Delete geojson file for next automated update
item_Delete = gis.content.get(geojson.id)
delete_result = item_Delete.delete()

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

AustinAverill — Wed, 16 Apr 2025 10:20:59 GMT

Gotcha, I wouldn't expect this script to take anymore than 5-10 minutes (depending on network performance on both yours and ESRI's side. Probably less.

Have you observed any of the print statements? If not, I would probably terminate the script manually and add in some logging. You could potentially be stuck trying to load imports or something else. Are you using an ArcGIS Pro created environment or have you made a specific venv?

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

JakeSkinner — Wed, 16 Apr 2025 13:03:43 GMT

Hi @MikeVolz, I see that you have the ability to run arcpy. I find arcpy's Append tool more robust. Instead of adding your GeoJSON file to Portal/AGOL, you could convert it to features using JSON to Features and then Append this feature class to your feature service.

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

MichaelVolz — Wed, 16 Apr 2025 13:19:07 GMT

Jake:

This was the original route I was taking, but I was trying to use the Truncate tool which is only available through ArcGIS API for Python so I had to log in twice, once for arcpy and then again for ArcGIS API for Python.

Can I use Delete Features tool to remove records from AGOL HFL before running the append tool? If so, is that tool fast to process?

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

JakeSkinner — Wed, 16 Apr 2025 16:58:36 GMT

@MichaelVolz yes, you can use the Delete Features tool, however you should note that the OBJECTIDs will not reset with this tool. This tool is pretty fast, but not as fast as truncate.

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

MikeVolz — Mon, 21 Apr 2025 14:39:25 GMT

Jake:

The script I attached did actually work, just not on its initial run. I also noticed that after the script ran successfully the first time some of the fields in the HFL that was truncated did not get all fields correctly populated with the Append process. I ran the script again and the fields were then populated so that was a false negative which I initially thought might be a bug.

I then added this script to a Scheduled Task and confirmed that it is a repeatable successful process where no field values are being lost in the Append process.

Also note that this was a small point data set, so not sure the same results will occur on a more complex polygon data set with a much larger number of features.

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

AustinAverill — Mon, 21 Apr 2025 14:59:32 GMT

Mike, this script should work well. The only issues that you may run into with field matching like described (that I can think of) is how the geojson is being generated. If you are passing this through a shpefile format at some point its possible that fieldnames are being altered, or if your field names don't match the hosted service to start with (spaces in column names, etc.) The Append function does a pretty good job of trying to recognize small discrepancies and reconcile them, but is far from a miracle worker!

Might be useful to save and examine the GEOJSON file on the next couple updates and make sure that the source file you are appending is matching 1:1 with the given dataset schema.

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

MikeVolz — Mon, 21 Apr 2025 15:11:35 GMT

Austin:

Based on previous experience with shapefiles truncating field names that were created on purpose with more than 10 characters for clarity, I avoid using shapefiles in processes such as this.

Also what software do you use to view a GEOJSON file to make it more human readable? Looking at GEOJSON in Google Chrome or Microsoft Edge is difficult.

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

AustinAverill — Mon, 21 Apr 2025 15:13:20 GMT

You can import it into Pro or Google Earth and see the field names and all pretty easily. Or you can just read the GEOJSON on using geopandas and examine the columns from the geodataframe.

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

sophered — Mon, 29 Dec 2025 16:35:35 GMT

Hello @AustinAverill !! I have a very large point file that ive been trying to test around with appending some of those features into a different layer as some what of a backup/storage, we're trying to automate and append of those features nightly into an agol notebook, i was wondering if you were accomplishing something similar and if so, how many of those features are you able to append? mine seems to cap at 28800 and i get a json decode error if i try to do any more than that. are you able to accomplish something similar in your environment

Re: Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

AustinAverill — Thu, 08 Jan 2026 11:51:27 GMT

Yes. I typically handle these workflows using a truncate and append approach, handling the geojson file to AGOL as an individual upload then using the hosted geojson file as my append data source. That code typically looks something like the following (Please note I am quickly adapting this from some classes I have written to handle this, so some of the syntax is going to look a bit odd):

from arcgis.gis import GIS from arcgis.gis import ItemProperties, ItemTypeEnum def upload_geojson(gis, filepath: str, itemname: str) -> str: if ".geojson" not in filepath: raise TypeError("Filepath does not indicate a .geojson format") folder = gis.content.folders.get() upProps = ItemProperties(title=f"{itemname}", description=f"Temp Upload for {itemname} append", item_type=ItemTypeEnum.GEOJSON) item_upload = folder.add(item_properties=upProps, file=filepath).result() return item_upload.id def clear_layer(hosted_fl: FeatureLayer) -> dict: try: hosted_fl.manager.truncate() output = { "status" : "success" } return output except Exception as e: output = { "status" : "error", "error" : str(e) } return output def append_data(hosted_fl: FeatureLayer, upload_itemid: str) -> dict: try: hosted_fl.manager.append(upload_itemid, "geojson") output = { "status" : "success" } return output except Exception as e: output = { "status" : "error", "error" : str(e) } return output gis = GIS() geojson = "path/to/your/file.geojson" hosted_fl = gis.content.get("itemid").layers[0] #or whatever ordinal position the layer is upload_itemid = upload_geojson(gis, geojson, "temp_upload") truncate_result = clear_layer(hosted_fl) if truncate_result["status"] == "success": append_result = append_data(hosted_fl, upload_itemid) print(append_result) if append_result["status"] == "success": #clear the geojson item to reduce storage costs temporary_item = gis.content.get(upload_itemid) temporary_item.delete()