topic Re: append new data instead of overwriting service (AGOL) in ArcGIS Online Questions

append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Fri, 01 Feb 2019 06:16:27 GMT

Hi folks,

I am looking for a way of appending new data to an existing feature service instead of overwriting it every time. The existing feature service holds attachments and becomes quite big over the time. Our internet here in Rwanda is sometimes not very stable so we are looking for a way to avoid uploading a GB of data every time the data is updated.

(By the way: Is there a way to see the size of a feature service including the attachments?) While locally I can append new data to an existing feature class I haven't seen this option yet for feature services. Would it maybe be possible to do this through a python script? I would be happy if you have any suggestions.

Re: append new data instead of overwriting service (AGOL)

KhaledHassen — Fri, 01 Feb 2019 16:36:10 GMT

Yes we do support append API in online feature service. Also it is available in python API.

Feature Service Append API

Thanks

Khaled Hassen

Re: append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Sun, 03 Feb 2019 07:09:20 GMT

An option that I have never seen before which makes it much easier is the Update function in the feature service with appending data from a gdb. I am not sure though if this makes a difference to the overwrite function out of ArcMap. Maybe someone can enlighten me on that?

I still like the idea of using the python API because I have the feeling to have more control about what is happening. However, I am new to the Python API but at least managed to get to the step where I have appended the data within the API with new_rows.append(old_fset.sdf) but the service is not updated. What would I need to do to have the changes actually applied to the feature service?

Thanks!

Re: append new data instead of overwriting service (AGOL)

KhaledHassen — Mon, 04 Feb 2019 15:49:24 GMT

You do not really need to do anything for the changes to be applied to the feature service after using append. If append succeeds, then new features will be appended and existing features will be updated (if you are using upsert). To update existing feature, you will need to use append matching key between the gdb feature class and the feature service layer. The online feature service layer field that is used as an append key needs to have a unique index in the online feature service layer.

The online home app contains update data option where you can update the online layer from different sources. Pl. check it out there and see if the data is updated with the options you have chosen. Let us know if this does not work for you.

Thanks

Khaled Hassen

Re: append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Mon, 04 Feb 2019 20:07:08 GMT

The strange thing in the python API is that the table view (SDF) shows that the data was appended and when I run the script again it it doesn't find any more missing features. But when I look in AGOL at the feature service data there are still only the first set of features. I can't figure out at which point to use the upsert. In general I don't need to update the existing features but it might be necessary in future. I have an unique identifier which is a string and which is not available when I use the append function from AGOL. Using AGOL I worry about two things:

1. If different users are using different copies from the feature class to add features I guess it could happen that an Object ID exists that is not the same feature as an Object ID among the new features. Thus it would be better to use our unique identifier.

2. I assume that when I upload a database with the feature class that it is effectively (from the data transfer) the same as updating/overwriting a service.

My next problem is that I have to find a solution without the python API - could I put the same in a python script running from ArcMap 10.6?

Re: append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Tue, 05 Feb 2019 05:45:12 GMT

I am afraid that using the append in AGOL will not be sustainable with the increasing amount of data over the time if I have to upload the gdb each time which is already for my small testing sample 33 MB. Our clients are looking for a good way to update the data without uploading the whole feature class each time, considering that the project might run over 10 years and they need to update the data regularly.

Re: append new data instead of overwriting service (AGOL)

KhaledHassen — Tue, 05 Feb 2019 15:24:47 GMT

>I have an unique identifier which is a string and which is not available when I use the append function from AGOL. >Using AGOL I worry about two things:

The key in the destination layer that you are using for upsert needs to have a unique index. This is why it does not show from the UX when appending from AGOL. The UX provides a way to add a unique index based on your unique field.

>If different users are using different copies from the feature class to add features I guess it could happen that an Object >ID exists that is not the same feature as an Object ID among the new features. Thus it would be better to use our >unique identifier.

Yes you cannot use the ObjectId for this. You have to either use a globalId or your own key that are the same in all copies. Copying data preserve within ArcGIS normally preserve GlobalIds, so duplicating data still reference the same feature.

>My next problem is that I have to find a solution without the python API - could I put the same in a python script running >from ArcMap 10.6?

You do not need to use Python API to do this. You can always use the feature service REST API to do this in any language. You can use fiddler to see how the home app uses the REST API to append data into online feature service.

Thanks

Khaled

Re: append new data instead of overwriting service (AGOL)

KhaledHassen — Tue, 05 Feb 2019 15:28:56 GMT

We have developed Append to deal with similar workflow like what you describe. We scale very well with the append API. The only other way to update the data which would definitely be more efficient to use sync and offline between online and the external system that updates the data. But this is not normally available specially if you are dealing with external data providers that use different system.

Thanks

Khaled Hassen

Re: append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Tue, 12 Feb 2019 08:13:44 GMT

Thanks Khaled for your input.

Do you maybe have a link to an example for me how to use the append API in python?

Re: append new data instead of overwriting service (AGOL)

KhaledHassen — Tue, 12 Feb 2019 18:00:48 GMT

Here is API doc help:

https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.features.toc.html?highlight=append#arcgis.features.FeatureLayer.append

Thanks

Khaled

Re: append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Wed, 13 Feb 2019 08:30:35 GMT

Thanks Khaled,

Forgive me my lack of knowledge but the link referes to the Python API for ArcGIS, right? There is no arcgis.gis module that I can use when trying to set up a python script tool for ArcMap 10.6, is it?

I need to achieve this somehow from ArcMap 10.6, ideally through a python script tool. But it's the first time that I try to access/work with AGOL data within python.

My idea is to download or read the table of the uploaded service, compare it to the updated version on my desktop, detect changes and new rows and then only append what is new or updated. I feel my brain is blocked to understand how I would do that in python...

Re: append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Fri, 15 Feb 2019 08:45:49 GMT

Soooo.... I am a big step further down the road now but now I am stuck.

(all done in one python script)

I ...

1. download the json of the existing AGOL feature service and write the UIDs in a list.

2. compare the UIDs of my desktop feature class with the ones in the list from 1.

3. select those items with UIDs that are not in the list and copy them to a new feature class in an otherwise empty gdb (copying the items keeps the attachments).

4. create a service definition draft (CreateMapSDdraft)

5. change a couple of settings like described here: CreateMapSDDraft—Help | ArcGIS for Desktop

6. stage the service and Upload the service definition

7. get the serviceItemID from the uploaded service

The idea was now to use the append API through python to append the new feature service to the old one but when I try it in the browser it says that Append is not enabled for my layer. I removed both sync and track changes settings. Using UpdateDefinition API adding Append to the capabilities {capabilities : "Create,Delete,Query,Update,Editing,Append", , it accepted my changes. I also checked for the {"supportsAppend" : true} setting which is correct for my service. It still keeps saying Append is not enabled.

Would you say in general I am on the right track? But how do I overcome the "Append not enabled" issue?

Since I saved the new data in a new gdb now I can use the append tool on the item details page but I would prefer to do it all in python. Any other hint?

Re: append new data instead of overwriting service (AGOL)

KhaledHassen — Fri, 15 Feb 2019 16:37:30 GMT

Append capability is not required if you use append as the owner or admin of the data.

As far as the capability goes, I think you would need to use the feature service updateDefinition API and not the layer updateDefinition API. This is normally the issue I have seen before. We should have error in the API but we are tolerant to the definition that is not valid.

I do not think you need to create SD file. Append accepts FileGDB and it is as good without the overhead of creating SD file with fileGDB inside.

Thanks

Khaled

Re: append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Fri, 15 Feb 2019 22:06:28 GMT

I gave up on appending the data from Python. I thought appending the new data from a gdb on the item details page would be good enough but testing it now I see that the attachments have not been appended with the points. What can I do?

Re: append new data instead of overwriting service (AGOL)

KhaledHassen — Fri, 15 Feb 2019 22:11:14 GMT

Yes sorry. Appending only for features. No schema or attachments.

Re: append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Sat, 16 Feb 2019 09:25:52 GMT

This is so disappointing 😞

do you have another suggestion how this could be achieved?

Re: append new data instead of overwriting service (AGOL)

KhaledHassen — Sat, 16 Feb 2019 14:35:40 GMT

Attachments in appending is something we are looking at to support in the next release or so for sure.

But for now, the only other solution besides overwrite the data with the new content in full is using sync.

So you will be working with a local copy of the data, edit the data and sync the changes (attachments + features).

You might check ArcMap create local copy for editing.

Using sync will only send the changes from your local copy and avoid sending data that has not changed. Sync will be even better than append since if you have low bandwidth, we will be sending only the changes.

Re: append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Mon, 18 Feb 2019 12:57:10 GMT

Thank you Khaled! I haven't thought of the sync option yet! Somehow I thought it would also not support attachments but it works like a charme!

So this is what I do now:

1. load the feature service into an MXD where I also have the local and updated version of the feature class.

2. Create a local copy of the feature service for editing on a very small extent which keeps the download small.

3. Run a script tool that

- downloads the json of the existing AGOL feature service and write the UIDs in a list.

- compares the UIDs of my desktop feature class with the ones in the list from above.

- select those items with UIDs that are not in the list and copy them to a new feature class in an otherwise empty gdb (copying the items keeps the attachments).

- use the geoprocessing tool append to append the new features to the old features

4. Sync with server version.

Awesome simple after wasting a lot of time trying to find a more AGOL focused solution...

Re: append new data instead of overwriting service (AGOL)

Katie_Clark — Mon, 18 Feb 2019 16:50:23 GMT

Is this a custom script that you wrote, or one that is widely available? Either way, would it be possible for you to share it here?

Many thanks!

Re: append new data instead of overwriting service (AGOL)

Nicole_Ueberschär — Tue, 19 Feb 2019 09:22:23 GMT

It is a custom script I wrote for a bigger project. I will send it as a private message.