Select to view content in your preferred language

Append Features to an AGOL Hosted Feature Layer (HFL) from Geojson File with ArcGIS API for Python - Very Slow

1897
12
Jump to solution
04-15-2025 10:59 AM
MikeVolz
Frequent Contributor

I am trying to append features to an AGOL HFL from a Geojson file with ArcGIS API for Python, but it is very slow.  I do not know if there is some sort of size limit for this procedure, as I am attempting to append 14K point features with about 20 columns to the AGOL HFL.

Has anyone attempted such an operation with success?

Any help or hints are greatly appreciated.  Thanks in advance.

0 Kudos
12 Replies
sophered
Regular Contributor

Hello @AustinAverill !! I have a very large point file that ive been trying to test around with appending some of those features into a different layer as some what of a backup/storage, we're trying to automate and append of those features nightly into an agol notebook, i was wondering if you were accomplishing something similar and if so, how many of those features are you able to append? mine seems to cap at 28800 and i get a json decode error if i try to do any more than that. are you able to accomplish something similar in your environment

0 Kudos
AustinAverill
Frequent Contributor

Yes. I typically handle these workflows using a truncate and append approach, handling the geojson file to AGOL as an individual upload then using the hosted geojson file as my append data source. That code typically looks something like the following (Please note I am quickly adapting this from some classes I have written to handle this, so some of the syntax is going to look a bit odd): 

from arcgis.gis import GIS
from arcgis.gis import ItemProperties, ItemTypeEnum


def upload_geojson(gis, filepath: str, itemname: str) -> str:
    
    if ".geojson" not in filepath:
        raise TypeError("Filepath does not indicate a .geojson format")

    folder = gis.content.folders.get()

    upProps = ItemProperties(title=f"{itemname}",
                                description=f"Temp Upload for {itemname} append",
                                item_type=ItemTypeEnum.GEOJSON)


    item_upload = folder.add(item_properties=upProps,
                                file=filepath).result()
    
    return item_upload.id

def clear_layer(hosted_fl: FeatureLayer) -> dict:
    try: 
        hosted_fl.manager.truncate()
        
        output = {
            "status" : "success"
        }
        return output
    
    except Exception as e:
        
        output = {
            "status" : "error",
            "error" : str(e)
        }
        
        return output
    
def append_data(hosted_fl: FeatureLayer, upload_itemid: str) -> dict:
    try:
        hosted_fl.manager.append(upload_itemid,
                                 "geojson")
        
        output = {
            "status" : "success"
        }
        return output
    
    except Exception as e:
        
        output = {
            "status" : "error",
            "error" : str(e)
        }
        
        return output
    
    
gis = GIS()
geojson = "path/to/your/file.geojson"
hosted_fl = gis.content.get("itemid").layers[0] #or whatever ordinal position the layer is

upload_itemid = upload_geojson(gis, geojson, "temp_upload")
truncate_result = clear_layer(hosted_fl)

if truncate_result["status"] == "success":
    append_result = append_data(hosted_fl, upload_itemid)
    print(append_result)
    
if append_result["status"] == "success":
    #clear the geojson item to reduce storage costs
    temporary_item = gis.content.get(upload_itemid)
    temporary_item.delete()

 

JakeSkinner
Esri Esteemed Contributor

@MichaelVolz yes, you can use the Delete Features tool, however you should note that the OBJECTIDs will not reset with this tool.  This tool is pretty fast, but not as fast as truncate.

0 Kudos