append new data instead of overwriting service (AGOL)

[Nicole_Ueberschär]

Hi folks,

I am looking for a way of appending new data to an existing feature service instead of overwriting it every time. The existing feature service holds attachments and becomes quite big over the time. Our internet here in Rwanda is sometimes not very stable so we are looking for a way to avoid uploading a GB of data every time the data is updated. 

(By the way: Is there a way to see the size of a feature service including the attachments?) While locally I can append new data to an existing feature class I haven't seen this option yet for feature services. Would it maybe be possible to do this through a python script? I would be happy if you have any suggestions. 

[KhaledHassen]

Yes we do support append API in online feature service. Also it is available in python API.

Feature Service Append API

Thanks

Khaled Hassen

[Nicole_Ueberschär]

An option that I have never seen before which makes it much easier is the Update function in the feature service with appending data from a gdb. I am not sure though if this makes a difference to the overwrite function out of ArcMap. Maybe someone can enlighten me on that?

I still like the idea of using the python API because I have the feeling to have more control about what is happening. However, I am new to the Python API but at least managed to get to the step where I have appended the data within the API with new_rows.append(old_fset.sdf) but the service is not updated. What would I need to do to have the changes actually applied to the feature service?

Thanks!

[KhaledHassen]

You do not really need to do anything for the changes to be applied to the feature service after using append. If append succeeds, then new features will be appended and existing features will be updated (if you are using upsert). To update existing feature, you will need to use append matching key between the gdb feature class and the feature service layer. The online feature service layer field that is used as an append key needs to have a unique index in the online feature service layer.

The online home app contains update data option where you can update the online layer from different sources. Pl. check it out there and see if the data is updated with the options you have chosen. Let us know if this does not work for you.

Thanks

Khaled Hassen

[Nicole_Ueberschär]

The strange thing in the python API is that the table view (SDF) shows that the data was appended and when I run the script again it it doesn't find any more missing features. But when I look in AGOL at the feature service data there are still only the first set of features. I can't figure out at which point to use the upsert. In general I don't need to update the existing features but it might be necessary in future. I have an unique identifier which is a string and which is not available when I use the append function from AGOL. Using AGOL I worry about two things: 

1. If different users are using different copies from the feature class to add features I guess it could happen that an Object ID exists that is not the same feature as an Object ID among the new features. Thus it would be better to use our unique identifier. 

2. I assume that when I upload a database with the feature class that it is effectively (from the data transfer) the same as updating/overwriting a service.

My next problem is that I have to find a solution without the python API - could I put the same in a python script running from ArcMap 10.6?  

[KhaledHassen]

>I have an unique identifier which is a string and which is not available when I use the append function from AGOL. >Using AGOL I worry about two things: 

The key in the destination layer that you are using for upsert needs to have a unique index. This is why it does not show from the UX when appending from AGOL. The UX provides a way to add a unique index based on your unique field.

>If different users are using different copies from the feature class to add features I guess it could happen that an Object >ID exists that is not the same feature as an Object ID among the new features. Thus it would be better to use our >unique identifier. 

Yes you cannot use the ObjectId for this. You have to either use a globalId or your own key that are the same in all copies. Copying data preserve within ArcGIS normally preserve GlobalIds, so duplicating data still reference the same feature.

>My next problem is that I have to find a solution without the python API - could I put the same in a python script running >from ArcMap 10.6?  

You do not need to use Python API to do this. You can always use the feature service REST API to do this in any language. You can use fiddler to see how the home app uses the REST API to append data into online feature service.

Thanks

Khaled

[Nicole_Ueberschär]

I am afraid that using the append in AGOL will not be sustainable with the increasing amount of data over the time if I have to upload the gdb each time which is already for my small testing sample 33 MB. Our clients are looking for a good way to update the data without uploading the whole feature class each time, considering that the project might run over 10 years and they need to update the data regularly. 

[KhaledHassen]

We have developed Append to deal with similar workflow like what you describe. We scale very well with the append API. The only other way to update the data which would definitely be more efficient to use sync and offline between online and the external system that updates the data. But this is not normally available specially if you are dealing with external data providers that use different system.  

Thanks

Khaled Hassen

[Nicole_Ueberschär]

Thanks Khaled for your input. 

Do you maybe have a link to an example for me how to use the append API in python?

[KhaledHassen]

Here is API doc help:

https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.features.toc.html?highlight=append#arcgi...

Thanks

Khaled