topic Re: Find Duplicates records in Python Questions

Find Duplicates records

CCWeedcontrol — Sat, 11 Dec 2021 15:06:04 GMT

I need to be able to add increments to duplicates in the P_ID field like below with arcpy. No sure how to though.

P_ID 23560 -->0

P_ID 23560 -->1

P_ID 23560 -->2

P_ID 23560 -->3

P_ID 23560 -->4

P_ID 23571 -->0

P_ID 23571 -->1

P_ID 23571 -->2.

I am working with the following code that adds 'Y' or 'No' in search of duplicates. Duplicates get a 'Y' and none Duplicates get 'N'.

import arcpy

#find Duplicate records, add Y or N in updateFeild

inShapefile = 'VacantLots'
checkField = "P_ID"
updateField = "Count"

with arcpy.da.SearchCursor(inShapefile, [checkField]) as rows:
    values = [r[0] for r in rows]

d = {}
for item in set(values):
    if values.count(item) > 1:
        d[item] = 'Y'
    else:
        d[item] = 'N'

with arcpy.da.UpdateCursor(inShapefile, [checkField, updateField]) as rows:
    for row in rows:
        if row[0] in d:
            row[1] = d[row[0]]
            rows.updateRow(row)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Find Duplicates records

JoshuaBixby — Tue, 09 May 2017 22:10:00 GMT

Several variations of this question have been asked in the past 6, maybe 12, months. I can't recall all of those other questions/discussions, but https://community.esri.com/thread/192311-find-duplicate-by-comparing-two-fields is one of them.

A quick question though. Since a group of records all share the same P_ID, how do you know which one is the original and which are the duplicates? Is the lowest ObjectID/FID the original?

Re: Find Duplicates records

RandyBurton — Tue, 09 May 2017 23:10:38 GMT

An old thread that might help: Identify duplicate records & increment

Re: Find Duplicates records

RandyBurton — Sat, 11 Dec 2021 15:06:07 GMT

This uses Arkadiusz Matoszka‌'s suggestion (from the link I posted above). It uses a dictionary in a manner similar to what you were trying. But it can run inside the UpdateCursor block, so you don't need the SearchCursor code. (Also, "Count" might not be a good field name as it may be an SQL reserved word.)

import arcpy

# find duplicate records and count them in dictionary

inShapefile = 'VacantLots'
checkField = "P_ID"
updateField = "fieldCount" # renamed field

d = {} # dictionary for counting

with arcpy.da.UpdateCursor(inShapefile, [checkField, updateField]) as rows:
    for row in rows:

        if row[0] not in d.keys():
            d[row[0]] = 0 # insert key into dictionary and set value to 0
            row[1] = 0 # value into updateField

        else:
            d[row[0]] += 1 #increment value in dictionary
            row[1] = str(d[row[0]]) # save value in updateField (str for text field)

        rows.updateRow(row)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Find Duplicates records

CCWeedcontrol — Thu, 11 May 2017 22:03:16 GMT

I tried the code you posted but everything in the filed was just '0'. I did look at the link you posted and I had something similar to this but i had the same results

Re: Find Duplicates records

CCWeedcontrol — Thu, 11 May 2017 22:28:44 GMT

Good point about which one is the original ID but for this case i don't need to actually know which is the original ID. It's a notification list to mail out notices and there is numerous double parcels. I need to remove the duplicates so when staff mails out letters their not sending out 3-7 letters to the same address.

I have ran into the case where people have requested every road name in our county. Our road center line is broken into numerous segments so i need to identify duplicates so this would be something i could use a number of ways.

Re: Find Duplicates records

RandyBurton — Fri, 12 May 2017 05:10:21 GMT

Are you working with a shape file in ArcMap's python window? If so, I think you need to be in an edit session for the shape file to be updated. Or are you working with a feature in a file geodatabase?

Re: Find Duplicates records

CCWeedcontrol — Fri, 12 May 2017 15:27:44 GMT

I tried it both ways i get the '0's'.

Re: Find Duplicates records

RandyBurton — Fri, 12 May 2017 15:57:43 GMT

Can you post your revised code?