Can you get rid of duplicate values in a csv file using ModelBuilder?

856
4
06-06-2019 05:50 AM
deleted-user-TdXMAAbco1KJ
New Contributor

I am currently interning and have created a model.  Part of the model will populate a csv file in excel, but when I run it again it does not delete previous values or get rid of duplicates.  I would like to know how I can delete duplicate values in the csv file?

0 Kudos
4 Replies
DuncanHornby
MVP Notable Contributor

This statement "...populate a csv file in excel" makes no sense, how can you have a CSV in an Excel document? You give zero details of the model, so this question is impossible to answer. If you write your data to a Table, so not Excel or CSV then you could use the Delete Identical tool...

TedKowal
Occasional Contributor III

I will take a pop shot at the question.....

I would run a python script that removes duplicate lines before you need to use the CVS .... Then your model would not have to worry about duplicate values.  In the example below, 1.csv contains duplicate lines; the script create a 2.csv that removes the duplicates.  You would use 2.csv in your model.

inFile = open('1.csv','r')

outFile = open('2.csv','w')

listLines = []

for line in inFile:

    if line in listLines:
        continue

    else:
        outFile.write(line)
        listLines.append(line)

outFile.close()

inFile.close()
curtvprice
MVP Esteemed Contributor

One could also wrap such a Python workflow in a function embedded in the Calculate Value tool. Just sayin!

0 Kudos
TedKowal
Occasional Contributor III

You could also run the script in the model.......  I would not recommend it.

0 Kudos