topic Re: Can you get rid of duplicate values in a csv file using ModelBuilder? in ModelBuilder Questions

Can you get rid of duplicate values in a csv file using ModelBuilder?

deleted-user-TdXMAAbco1KJ — Thu, 06 Jun 2019 12:50:56 GMT

I am currently interning and have created a model. Part of the model will populate a csv file in excel, but when I run it again it does not delete previous values or get rid of duplicates. I would like to know how I can delete duplicate values in the csv file?

Re: Can you get rid of duplicate values in a csv file using ModelBuilder?

DuncanHornby — Wed, 12 Jun 2019 09:02:12 GMT

This statement "...populate a csv file in excel" makes no sense, how can you have a CSV in an Excel document? You give zero details of the model, so this question is impossible to answer. If you write your data to a Table, so not Excel or CSV then you could use the Delete Identical tool...

Re: Can you get rid of duplicate values in a csv file using ModelBuilder?

TedKowal — Sun, 12 Dec 2021 08:48:21 GMT

I will take a pop shot at the question.....

I would run a python script that removes duplicate lines before you need to use the CVS .... Then your model would not have to worry about duplicate values. In the example below, 1.csv contains duplicate lines; the script create a 2.csv that removes the duplicates. You would use 2.csv in your model.

inFile = open('1.csv','r')

outFile = open('2.csv','w')

listLines = []

for line in inFile:

    if line in listLines:
        continue

    else:
        outFile.write(line)
        listLines.append(line)

outFile.close()

inFile.close()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Can you get rid of duplicate values in a csv file using ModelBuilder?

curtvprice — Wed, 12 Jun 2019 23:20:25 GMT

One could also wrap such a Python workflow in a function embedded in the Calculate Value tool. Just sayin!

Re: Can you get rid of duplicate values in a csv file using ModelBuilder?

TedKowal — Thu, 13 Jun 2019 15:34:15 GMT

You could also run the script in the model....... I would not recommend it.