append new data instead of overwriting service (AGOL)

[Nicole_Ueberschär]

Hi folks,

I am looking for a way of appending new data to an existing feature service instead of overwriting it every time. The existing feature service holds attachments and becomes quite big over the time. Our internet here in Rwanda is sometimes not very stable so we are looking for a way to avoid uploading a GB of data every time the data is updated. 

(By the way: Is there a way to see the size of a feature service including the attachments?) While locally I can append new data to an existing feature class I haven't seen this option yet for feature services. Would it maybe be possible to do this through a python script? I would be happy if you have any suggestions. 

[Nicole_Ueberschär]

Thanks Khaled, 

Forgive me my lack of knowledge but the link referes to the Python API for ArcGIS, right? There is no arcgis.gis module that I can use when trying to set up a python script tool for ArcMap 10.6, is it?

I need to achieve this somehow from ArcMap 10.6, ideally through a python script tool. But it's the first time that I try to access/work with AGOL data within python. 

My idea is to download or read the table of the uploaded service, compare it to the updated version on my desktop, detect changes and new rows and then only append what is new or updated. I feel my brain is blocked to understand how I would do that in python...

[Nicole_Ueberschär]

Soooo.... I am a big step further down the road now but now I am stuck.

(all done in one python script)

I ...

1. download the json of the existing AGOL feature service and write the UIDs in a list.

2. compare the UIDs of my desktop feature class with the ones in the list from 1.

3. select those items with UIDs that are not in the list and copy them to a new feature class in an otherwise empty gdb (copying the items keeps the attachments).

4. create a service definition draft (CreateMapSDdraft)

5. change a couple of settings like described here: CreateMapSDDraft—Help | ArcGIS for Desktop 

6. stage the service and Upload the service definition

7. get the serviceItemID from the uploaded service

The idea was now to use the append API through python to append the new feature service to the old one but when I try it in the browser it says that Append is not enabled for my layer. I removed both sync and track changes settings. Using UpdateDefinition API adding Append to the capabilities {capabilities : "Create,Delete,Query,Update,Editing,Append", , it accepted my changes. I also checked for the {"supportsAppend" : true} setting which is correct for my service. It still keeps saying Append is not enabled. 

Would you say in general I am on the right track? But how do I overcome the "Append not enabled" issue?

Since I saved the new data in a new gdb now I can use the append tool on the item details page but I would prefer to do it all in python. Any other hint?

[KhaledHassen]

Append capability is not required if you use append as the owner or admin of the data.

As far as the capability goes, I think you would need to use the feature service updateDefinition API and not the layer updateDefinition API. This is normally the issue I have seen before. We should have error in the API but we are tolerant to the definition that is not valid.

I do not think you need to create SD file. Append accepts FileGDB and it is as good without the overhead of creating SD file with fileGDB inside.

Thanks

Khaled

[Nicole_Ueberschär]

I gave up on appending the data from Python. I thought appending the new data from a gdb on the item details page would be good enough but testing it now I see that the attachments have not been appended with the points. What can I do?

[KhaledHassen]

Yes sorry. Appending only for features. No schema or attachments.

[Nicole_Ueberschär]

This is so disappointing 😞

do you have another suggestion how this could be achieved?

[KhaledHassen · Accepted solution]

Attachments in appending is something we are looking at to support in the next release or so for sure.

But for now, the only other solution besides overwrite the data with the new content in full is using sync.

So you will be working with a local copy of the data, edit the data and sync the changes (attachments + features).

You might check ArcMap create local copy for editing.

Using sync will only send the changes from your local copy and avoid sending data that has not changed. Sync will be even better than append since if you have low bandwidth, we will be sending only the changes.

[Nicole_Ueberschär]

Thank you Khaled! I haven't thought of the sync option yet! Somehow I thought it would also not support attachments but it works like a charme!

So this is what I do now: 

1. load the feature service into an MXD where I also have the local and updated version of the feature class.

2. Create a local copy of the feature service for editing on a very small extent which keeps the download small. 

3. Run a script tool that 

    - downloads the json of the existing AGOL feature service and write the UIDs in a list.

    - compares the UIDs of my desktop feature class with the ones in the list from above.

    - select those items with UIDs that are not in the list and copy them to a new feature class in an otherwise empty gdb (copying the items keeps the attachments).

    - use the geoprocessing tool append to append the new features to the old features

4. Sync with server version. 

Awesome simple after wasting a lot of time trying to find a more AGOL focused solution...

[Katie_Clark]

Is this a custom script that you wrote, or one that is widely available? Either way, would it be possible for you to share it here?

Many thanks!

[Nicole_Ueberschär]

It is a custom script I wrote for a bigger project. I will send it as a private message.