Updating large hosted feature and table layers

Question

Hello community,

I am currently trying to find a way to update a hosted feature table that has 1.7M records (20 table fields) and a hosted feature layer (Points) that has just over 500k records in AGOL.  The two layers are being used in an ExB and need to be updated weekly.  I am trying to use the Truncate and Append REST services to assist in this and have tried both Python and FME to run the updates.

The Truncate works perfectly and the only issue that I have is with the Append.

For the FME process I use the HTTPCaller to call the two REST services after I upload a zipped FGDB to AGOL.  This works to a point but I usually get just a partial update most likely due to the gateway timeout.  When it works, it takes about 20 minutes.

For the Python process I break the FGDB up into 50K record chunks, upload it, append it and then call the next chunk.  The results for my 1.7M record table are all over the place.  Sometimes I get zero records updated, sometimes I get to the 7th chunk (out of ~36) and sometimes I get to the point where it only has 4000 records left.  It makes no sense.  The entire Python process has only worked once out of about 20 tries and the FME process worked once out of about 10 tries.

I also have ArcGIS Enterprise and the entire process takes about 1h20m hour to run using a normal truncate and append in FME using a feature writer.  We use Enterprise for internal use.

I've attached my Python as it does work, but I am trying to find a way to automate and efficiently update my data weekly without running into what I assume is the gateway timeout.  I also attached a screenshot of the processing where I can tell that it hits the timeout.

Thank you for any help that you can give,

Peter

stations_update.py

Screenshot 2026-08-12 100727.png

PeterHearns_Canada · Answer

I wish that I could do this, but the data is fully re-uploaded once a month by many partners.  Unique IDs are not included even though it would be a possibility, but would have to be managed by the 100s of individual partners. I'll be looking into this, but not much hope for it.

I tried 25k down to 10K chunks yesterday and it failed all day long.  At that point it takes almost as much time as the native FME FeatureWriter and the FeatureWriter is definitely reliable and takes just under 2 hours to do a full 1.7M record refresh. 2 hours plus my sanity is worth it.

I'm guessing that my only other option is the Premium data store.

joshkalovcook · Answer

Do you actually need to do a full truncate and replace with each update? What are the data updates like? Are most or all of the 1.7M records regularly getting updated or is it just some records? Or is it mostly new records and not edits of existing records? Are records regularly deleted or is that an infrequent occurrence?

You can do comparisons between your new data and the hosted data and only update, append, delete the records needed. How that comparison happens could depend a lot on the nature of change and the format of data. Is there a unique id field? Is there an editor tracking or similar update timestamp field? If so, you could query the hosted data for just those 2 fields (or whatever fields you want to do the comparison on if there isn't an update timestamp field). Then compare them against your new data to identify adds, updates, deletes and only make those changes to the hosted layers.

The benefit of a comparison and upsert + delete route depends a lot on the expected quantity of record changes each week. If 1.5 million records are changing each week, it may not help much. But if its only 10 thousand or 50 thousand, that significantly cuts down what you actually need to upload.

For what it's worth, after a lot of trial and error and tweaks, I've had a script running successfully daily to update a hosted feature layer using ArcGIS Python API layer.edit_features() sent in batches of 200 records. It's been a few years since I have tested higher batch counts so that might be unnecessarily low.

PeterHearns_Canada · Answer

Hi Tom,

I definitely appreciate the ideas.

I tried using 25k chunks and it worked... the first time ... and then failed on chunk 23 an hour later.  I'll be testing at various times throughout the day.

We've contacted out Esri rep and will be submitting to support shortly.

I was thinking about the distributed collab between our AGE and AGOL, but the end result is a public facing ExB and I am hesitant to making the data on the AGE side public... but I don't have experience with this, so maybe it is an option.

Peter