Duplicates created during Append Tool (From ArcGIS Online to ArcGIS Pro)

[EKandD]

Hello,

I am trying to update my layers in ArcGIS Pro with new data gathered in ArcGIS Online by using the Append tool in model builder. I have 70 different layers in ArcGIS Pro that I need to update regularly, each with anywhere from five to hundreds of rows/data points in them. I tried to create a Model Builder that would use the Append tool to add data from the ArcGIS Online Layers to the ArcGIS Pro layers. The layers in ArcGIS Online have the same rows/data as the layers in ArcGIS Pro, but will often get one or two new rows/data points (from workers who use Field Maps), which have to be added to the layers in ArcGIS Pro. I have been doing this manually, which usually takes a full 8 hour work day to do all the merging/renaming/deleting, so I would like to modify this process to be automated.

In the screenshots below, I am showing a small portion of my model.  In the Append tool, I have the ArcGIS Online layer as the Input Dataset, my ArcGIS Pro layer as my Target Dataset, the Field Matching Type set to "Use field map to reconcile differences", and in the Environments Tab I selected "Preserve Global IDs".

I am unsure of how to append only the new data gathered from ArcGIS Online and add it to the ArcGIS Pro layers. When I append all of the data (because the ArcGIS Online and ArcGIS Pro layers have a lot of the same data) the Target Dataset in ArcGIS Pro now has a table full of duplicates and one new row of data.

I tried to use the "Delete Identical" tool to delete any identical rows/data, but it will not let me select the Global ID field as a field to find duplicates of. That is the only field in all of my attribute tables that is unique to each data point. I ran into the same issue when I tried the Find Identical tool.

Questions:

1. Is there a way to be able to add the Global ID field to the Delete Identical tool field?

2. Is there a better way to automatically set up the Append tool, so that only new data from the ArcGIS Online layer is Appended to the ArcGIS Pro layers, stopping all of the duplicate data from being added?

I have been struggling with this for a while now, any help is greatly appreciated.

Thanks!

[dgiersz_cuyahoga]

You could do a Select By Attributes to find only the new features using a created by date. Or you can use the Delete Rows tool on your target to empty it out first, then Append all the records.

The first option is probably better if you have a lot of features, since the Append will take longer depending on the size of the input.

[JesseWickizer]

If you have a unique identifier from the source data, use the Matching Fields for Update parameters of the Append tool to update existing records in the target data instead of adding them to the target.