topic Re: Finding Duplicates in Python Questions

Finding Duplicates

JasonBoothe — Thu, 03 May 2012 17:08:40 GMT

Hey folks, I don't mean to beat a dead horse being that i have seen this discussed before, but here is my issue. I am having to merge two sets of property records, the records overlap for about 10 months worth, so i need to weed through them find the duplicate records, verify they are duplicates, and correct accordingly. I have seen code on here that will mark in a designate field all occurrences of a duplicate record with a designated string, however i have not been able to get that python script to work (4021 ArcGIS 10 Pre Release finding duplicates). So in the mean time i am left using Knowledge base solution 31854 which works great if my dupes came only in sets of 2. So any other ideas, or am i missing something here. And just a FYI, i am very green when it comes to Python and scripting in ArcGIS outside of Find Attribute Queries. Thanks

Re: Finding Duplicates

JoeFlannery — Thu, 03 May 2012 17:21:13 GMT

Have you discovered Find Identical and Delete Identical in the Data Management toolbox?

Re: Finding Duplicates

JasonBoothe — Mon, 07 May 2012 12:30:10 GMT

I have, unfortunately my use of it is restricted due to a licensing issue beyond my control.

Re: Finding Duplicates

StephenKruzik — Mon, 07 May 2012 14:27:59 GMT

taken from: http://forums.arcgis.com/threads/4021-ArcGIS-10-Pre-Release-finding-duplicates

As long as you have the tables joined together so that everything is one dataset...

Calculate field, and choose the python parser and in the pre-logic:

[HTML]uniqueList = []
def isDuplicate(inValue):
if inValue in uniqueList:
    return 1
else:
    uniqueList.append(inValue)
    return 0[/HTML]

then in the calculate field:

[HTML]isDuplicate(!FIELD_NAME!)[/HTML]

Re: Finding Duplicates

JasonBoothe — Wed, 16 May 2012 18:33:15 GMT

Thanks Stephen,

This sort of works for what i need it to do, i'll just have to add an additional step in the process to finalize verification of the records prior to removal.

Re: Finding Duplicates

NobbirAhmed — Wed, 16 May 2012 20:11:20 GMT

The licensing restriction is loosened a bit - if you use one attribute field at a time (except for Shape field) then Find Identical tool works on all license levels.