Appending- how to avoid duplicates?

[TheoFaull]

I have two point shapefiles, both have the exact same fields in them. There are some new records AND some duplciate records when comparing the two datasets.

I want to use the append tool as I don't want to create a new dataset, I just want to add data to the existing original shapefile.

However, when I append the two shapefiles, matching records are appended, thus leaving lots of duplicate records. How can I tell my script to only append new records and ignore duplicates?

[DanPatterson_Retired]

If they are truly identical.... Find identical

[JayantaPoddar]

and Delete Identical.

[TheoFaull]

That's only available with an advanced license though. I only have Basic

And yes most of the records are duplicates. Maybe there is away to only append records that are new...

[JayantaPoddar]

Add XY coordinates to your featureclass, then Dissolve on the X,Y fields.

[TheoFaull]

There are multiple seperate points that share the exact same coordinates. So dissolving based on that would combine points together that are infact independent of each other.

There is a unique ID field however... I could dissolve the appended dataset based on that field?

[TheoFaull]

Dissolve won't work as it create a new dataset. It's crucial to me that the final dataset, is the existing one. (Lots of ArcMap MXDs link to that shapefile.)

[TheoFaull]

Also when appending the new dataset to the original, I want the new dataset to take priority over the existing one. ie. if two records match, then old records are overwritten by the new ones.

[BruceHarold]

The Data Interoperability extension has a couple of functions for this use case, the ChangeDetector and Matcher transformers.  The former identifies added, deleted and unchanged features between two datasets, the latter finds matching features.  Both allow you to choose the combination of geometry and attribute tests you need.  If you can share your data I can build a sample, either in this thread or via email.  I would need to know your preference of 10.5 or Pro 1.4.1 tool version.

[TheoFaull]

Thanks Bruce but the data contains 1000s of records containing confidential tenant information. I doubt I'd be allowed to share

In an ideal world, I just need a simple script that deletes the existing dataset and places the new, updated dataset in the same directory and with the same name. That way all the MXDs won't need to be re-sourced each time.

The reason I can't do this method is because of an ever present LOCK file on the existing dataset (it's under constant access by our GIS server...).

Maybe a script that says:

1. Stop Server GIS service

2. Delete Points.shp

3. Create updated version of Points.shp in same directory with same name.

4. Start Server GIS service