Select to view content in your preferred language

Feature size increasing after truncate and append

499
3
02-12-2026 02:28 PM
Labels (2)
twangtx
Occasional Contributor

Hi Everybody,

I've encountered some inexplicable result when I'm running daily/weekly python scripts to updated hosted feature layers in ArcGIS Online where the feature storage size seems to balloon in size after each truncate and append option. The hosted feature layer started a few weeks ago at like 60mb of feature storage, but it's now 1.5gbs and I'm at a loss at what's happening.

The layer is sync and edit enabled.

Basically we have two methods of updating feature layers.

1. Is where we call rest endpoint directly and perform arcpy delate rows and append. And it seems to work and the end user doesn't seem to notice any issues.

HFL URL = "[URL]"
arcpy.DeleteRows_management(HFL URL)
arcpy.Append_management(Local FC Source, HFL URL,"NO_TEST")
2. The other method, we zip up a file geodatabase, upload it to AGO and perform a truncate and append (from the fgdb) to the feature layer.
 
Below are the steps after the zipped fgdb gets uploaded to AGO.
 
flc.manager.update_definition({"capabilities": "Create,Delete,Query,Update,Editing,Sync", "syncEnabled": False})
featLyr.manager.truncate()
flc.manager.update_definition({"capabilities": "Create,Delete,Query,Update,Editing,Sync", "syncEnabled": True})
featLyr.append(item_id=fgd_item.id,
                    upload_format="filegdb",
                    upsert=False,
                    field_mappings=[])

We also have scripts that just item.publish(overwrite=True), but it's a little more finicky I've found to use than the truncate and append, until now. But I've not noticed feature layers exploding in storage space.

I almost feel like there should be like there's an adds and deletes table that needs to be compressed or something in AGO. 

Has anyone encountered this issue or have any insights?

Thank you,

Thomas

0 Kudos
3 Replies
AaronKoelker
Frequent Contributor

I know you're disabling it mid-step, but can you test it with a duplicate feature layer that doesn't have sync enabled at all? Sounds like it is still storing previous versions or a history of previous edits somewhere, and if you're doing a full truncate/append that's a lot of "edits". 

When I need to do full overwrites I either use the .update() method to replace the whole file, or use spatially enabled dataframes (SEDF) and .edit_features(adds='your SEDF'). Something like the examples below. Could try one of these as an alternative to append() . 

### Update Hosted Feature Layers Examples
from arcgis.gis import GIS
from arcgis.features import FeatureLayerCollection

updated_data = r'this_is_a_path_to_file.csv' # path to your updated dataset, the file type must match whatever the service was published with
itemid = '' # the itemid for the hosted feature service you want to update in ArcGIS Online, you can get it off item page in AGO
portal_url = '' # "https://example.maps.arcgis.com"
user = '' # AGO username
pw = '' # AGO password


### Log in to ArcGIS Online account
gis = GIS(portal_url, user, pw)
un = gis.properties.user.username
print(f'\nLogged in as {un} on {portal_url} \n')


#############################################################################
### Example 1 (Overwrite directly from local file)
# If you published to AGO from a local file (csv, fgdb, etc.), this is the simplest way to overwrite
# The new data must have the same filename and file type, and FGDBs must be zipped
hostedfeature = gis.content.get(itemid)
flayer = FeatureLayerCollection.fromitem(hostedfeature)
flayer.manager.overwrite(updated_data)
print("Hosted feature layer updated with the latest and greatest")

# That is the barebones method, but you can update lots of other stuff too, for example the description, title, etc.
# https://developers.arcgis.com/rest/users-groups-and-items/common-parameters/
item_properties={
    "description":"Here is my description, isn't it great",
    "title":"Great Item Title"
    }
hostedfeature.update(item_properties)

# Or the thumbnail
thumb = r'path_to_some_picture.jpg' # can also use a image url
hostedfeature.update(thumbnail=thumb)

# Or you can disable and enable settings, like Sync
syncoff = {"capabilities": "Query, Extract"} 
hostedfeature.manager.update_definition(syncoff)

syncon = {"capabilities": "Query, Extract, Sync"}
hostedfeature.manager.update_definition(syncon)

# https://developers.arcgis.com/python/latest/samples/overwriting-feature-layers/#:~:text=updated_capitals_csv%27%2C%20csv_file_name))-,Overwrite%20the%20feature%20layer,-Let%20us%20overwrite


#############################################################################
### Example 2 (Go the edit route with a full truncate and append)
# You can do this either from a regular pandas dataframe if it's a table, a spatially enabled dataframe if there is geometry, or from a featureset
# If you are adding more than 1000 features you want to chunk it out
import pandas as pd
import os

hostedfeature = gis.content.get(itemid).layers[0] # if you have a hosted table you can use .tables[0] instead
hostedfeature.manager.truncate()

local_featureclass = os.path.join(r"path to a file geodatabase .gdb", "MyFeatureClass")
sedf = pd.DataFrame.spatial.from_featureclass(local_featureclass)

n = 1000
for i in range(0, len(sedf), n):
    chunk = sedf[i:i+n]
    result = hostedfeature.edit_features(adds=chunk)
    print(f'added items {i+1}-{i+n}...')

 

-Aaron
0 Kudos
twangtx
Occasional Contributor

Hi Aaron,

Thank you for responding.

I think we've got the issue "resolved" in that we kind of just went with the nuclear option. We ended up just overwriting the hosted feature layer from Pro with a feature class of the same schema. And the method that has been working for us so far is:

  1. Zip up a file geodatabase 
  2. Upload to AGO
  3. Disable sync
  4. Truncate
  5. Append from file geodatabase
  6. Reenable sync
  7. Delete the file geodatabase in AGO.

I think maybe the critical thing we did was append before reenabling the sync.

It's been running daily for a few weeks, and the feature storage size has been staying consistent.

Thank you,

Thomas

0 Kudos
AaronKoelker
Frequent Contributor

Ah that makes sense, I didn't catch that in the original code you shared. Moving the reenable sync step to after the append was almost certainly it. Glad you were able to figure it out. 

-Aaron
0 Kudos