topic Mark duplicate features in attribute table with Field Calculator, update to Arc10 in Python Questions

Mark duplicate features in attribute table with Field Calculator, update to Arc10

ChrisWoodward1 — Tue, 06 Sep 2011 01:39:48 GMT

I've got a handy piece of VB script that I used to routinely run on Arc9.3 to mark duplicate features in an attribute table based upon unique ID's in the table. I typically use this on repeat points that appear within a dataset more than once (based upon ID value, Date/Time and position). The problem is, the script won't work in Arc10 and I wasn't the original author so I'm not sure how to update it.

I'm running Arc10 SP2 on a Windows 7 and/or XP machine. My typical workflow is to concatenate a unique ID 'Unique_ID' (text, 255) based upon several attributes (say ID, Date/Time, Lat./Long). I then create a attribute called 'dup' of a short interger type. I run the field calculator on the dup field and load the following script and what it would normally do is to mark duplicate values of the Unique_ID value with a 1 value. Both records wouldn't be marked, only the second or greater duplicate Unique_ID value record.

Within the field calculator, I'd have the "Show Codeblock" checked and the following in the "Pre-Logic Script Code" area:

Static d As Object
Static i As Long
Dim iDup As Integer
Dim sField
'========================
'Replace the field name bellow
sField = [Chris_ID]
'========================
If (i = 0) Then
Set d = CreateObject("Scripting.Dictionary")
End If
If (d.Exists(CStr(sField))) Then
iDup = 1
Else
d.Add CStr(sField), 1
iDup = 0
End If
i = i + 1

The following would also be in the bottom part: Dup=iDup

If someone could help me with updating this VB script for Arc10, it would be greatly appreciated!

Thanks,
Chris

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

StacyRendall1 — Sat, 11 Dec 2021 23:15:19 GMT

Here is a Python solution. At the moment it returns None for the first value (although you can change this to 0 if you want) and it counts up the duplicates, so the first duplicate is 1, the second is 2 and so on (but you can change that too - just make the line read return 1 instead of return d[inVal])

Pre-Logic Script Code:
d = {} # creates an empty dictionary the first time

def findDuplicates(inVal):
 try:
  d[inVal] += 1 # works only if this value has been encountered before (is a valid key)
  return d[inVal] # returns the number, you could just make this return 1
 except KeyError:
  d[inVal] = 0 # if this is the first time, highlight it
  return None # or 0...

Expression: dup =
findDuplicates(!Unique_ID!)

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

ChrisWoodward1 — Wed, 14 Sep 2011 17:06:20 GMT

Stacey,

Thanks so much for your help. I've tried running the script in the field calculator but get an error message "A field name was not found or there were unbalanced quotation marks". Am I doing something wrong? I've copied your script precisely as you have it here for the Pre-Logic Script Code and the Expressionn dup = portion of the script. Thanks.

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

MarcNakleh — Sat, 11 Dec 2021 23:15:21 GMT

Stacy: Just thinking about it, I think a more Pythonic way would be to use a dictionary's built-in setdefault method, which returns a default value if the key isn't found (which works well with cwoodward's issue, I think.)

Pre-Logic Script Code:
d = {} # creates an empty dictionary the first time

def find_duplicates(in_val):
 d[val] = d.setdefault(val, -1) + 1
 return d[val]

Expression: dup =
find_duplicates(!Unique_ID!)

Though I find yours more legible!

cwoodward's : I think you need to replace the expression:

findDuplicates(!Unique_ID!

with the expression:

findDuplicates(!Chris_ID!)

if the column with the values you are checking for duplicates is called 'Chris_ID'. In the Python Field Calculator, values surrounded by exclamation marks refer to column names (just like brackets [] in VB code.)

Hope this helps!

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

StacyRendall1 — Sat, 11 Dec 2021 23:15:24 GMT

Marc, that's impressive! I haven't come across setdefault before, but I can see where it might be handy.

Just noticed that on the module line you had in_val, but the rest of the module refers to val. cwoodward, this should work (see attached picture for where to put things; resultsField is the field the results should go in, numerator is the field where the duplicates are):

Pre-Logic Script Code:
d = {} # creates an empty dictionary the first time

def find_duplicates(val):
 d[val] = d.setdefault(val, -1) + 1
 return d[val]

Expression: dup =
find_duplicates(!Unique_ID!)

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

MarcNakleh — Thu, 15 Sep 2011 11:27:04 GMT

Hi Stacy,

Thanks for the correction!

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

GeorgeNewbury — Sat, 11 Dec 2021 23:15:27 GMT

Thanks for posting that. I modified it to find duplicate points based on same exact X,Y:

Pre-Logic Script Code:
d = {} # creates an empty dictionary the first time

def find_duplicates(valX,valY):
 d[(valX,valY)] = d.setdefault((valX,valY), -1) + 1
 return d[(valX,valY)]

Expression: dup =
find_duplicates( !SHAPE.firstpoint.x!, !SHAPE.firstpoint.y! )

I used the 'Feature Vertices to Points' tool, but it produced duplicate points at the common vertices. I couldn't find an out of the box tool to filter them.

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

RobBlash — Tue, 12 Jun 2012 11:58:27 GMT

Is there a way to modify this script to simply select the features that are duplicated? It would be helpful to locate dups without the need to add a new field.

Thanks,

Rob

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

MarcinGasior — Tue, 12 Jun 2012 12:13:26 GMT

Maybe Find Identical tool will help?

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

RobBlash — Tue, 12 Jun 2012 12:40:10 GMT

Thanks, I'll take a look. I was hoping to come up with a solution available at all license levels.

Pre-10 we had a button to bring up a form, choose the field, and select duplicates. I'd like to get that functionality back in 10.1 but of course without VBA.

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

ChrisGraves — Wed, 19 Sep 2012 05:57:09 GMT

Hey GIS Users,

I'm trying to identify duplicate records from a field and then change these fields so they become unique (eg A,B,C, or 1,2,3) and assign this to the end of the text (eg F3241 A56_A or F3241 A56_1).
I've tried the scripts that are in this forum but can't get these to work or not sure if they will give me the result I'm after.
Any input into to this would be greatly appreciated.

Cheers,

Chris

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

shivamparashari — Wed, 05 Feb 2014 03:48:26 GMT

Hi,

Actually, I was working with Address Points and had to calculate Spatial Duplicates(same X & Y Coordinates) and Address Duplicates(same attributes). gnewburyiv gave the idea and it worked. Gracias. Now, for address duplicates, I need to check for more than two fields(valX, valY, valZ, valA, valB etc.). What can be done in this case. (:

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

RobBlash — Wed, 05 Feb 2014 09:32:20 GMT

The old "find duplicates" code I was using at 9.x was ported as an addin, thanks to whomever did this work.

http://www.arcgis.com/home/item.html?id=2f8c0a2477db465b8ca2a9f7f519eb5a

It works great to find attribute duplicates and doesn't require new fields, editing, info license etc.

Hi,

Actually, I was working with Address Points and had to calculate Spatial Duplicates(same X & Y Coordinates) and Address Duplicates(same attributes). gnewburyiv gave the idea and it worked. Gracias. Now, for address duplicates, I need to check for more than two fields(valX, valY, valZ, valA, valB etc.). What can be done in this case. (:

To find duplicates amongst many fields you can concatenate the values into one field and then check with the tool I linked above. If you're looking for spatial duplicates in points try using the point distance in toolbox (I think you need an editor or an info license)

Re: Mark duplicate features in attribute table with Field Calculator, update to Arc10

shivamparashari — Wed, 05 Feb 2014 10:00:14 GMT

Hi rblash Thanks for the reply. Actually, I am working with ArcGIS 10.0, so the add-in is not working.
I did this thing in Python-

pre-logic script code:

def find_duplicates (valA , valB , valC , valD , valE , valF , valG):
d[( valA , valB , valC , valD , valE , valF , valG)] = d.setdefault ( ( valA , valB , valC , valD , valE , valF , valG) , -1) +1
return d[ ( valA , valB , valC , valD , valE , valF , valG) ]

ObjectID:

find_duplicates ( !hnr1! , !pc_main! , !city! , !attribdate! , !geocodinga! , !AdminPlace! , !Building_9! )
(: