topic Re: Count Rows being processed with INSERT Cursor in Python Questions

Count Rows being processed with INSERT Cursor

jaykapalczynski — Sat, 11 Dec 2021 06:20:52 GMT

I have a script that is looping through many txt files and adding them to a feature class. I want to tally the total number of records added.

I am trying to use a RecordCounter Variable to count the records....It DOES work to some degree...it count the first txt file that is being processed but if I process 7 files it only gives me the number from the first txt file.

Any thoughts?

# .... SNIP

else: # SEEING THERE ARE NO DUPS PROCEED WITH PROCESSING
    print "They are no dups so processing is continuing"
    #Read through each line of the csv, but skip the first row because it is the header
    #The count variable skips of the first row
    count = 0
    RecordCounter = 0
    for ln in open (textFile, 'r').readlines():  
        if count > 0:
            #Each line in the csv is a string, so turn it into a list so you can reference each column
            lineSplit = ln.split(",")

            #Use index positions to get the right column from the csv
            #Make sure the data is the correct type (i.e. X and Y need to be numbers, not strings)
            CollarSeri = long(lineSplit[0])
            Year = str(lineSplit[1])
            Julianday = str(lineSplit[2])                               
 
            #Insert the values from csv into feature class
            #The order of the fields here, matches the order of the fields in line 4
            cursor.insertRow ([CollarSeri, Year, Julianday, Hour, Minute, Activity, Temperature, Latitude, Longitude, HDOP, NumSats, FixTime, Date, _2D_3D, BearID, getTime3, getDateTime, FileName, shapeVal])
                
         RecordCounter = RecordCounter + 1

         count += 1
                                   
     del cursor
     
     print RecordCounter‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

# .... SNIP‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Count Rows being processed with INSERT Cursor

jaykapalczynski — Sat, 11 Dec 2021 06:20:54 GMT

I did it in another area...all set

def file_len(fname):
    with open("C:/Users/NewFiles/" + fname) as f:
        for i, l in enumerate(f):
            pass
    return i + 1

       for file in os.listdir(dir_src):
           #Get the number of records being processed calling function file_len below
           testfile = str(file)
           teststr = file_len(file)
                
           # Count how many Files are being processed.
           FileCounter = FileCounter + 1
                
           # Count the Records being processed each time
           TotalRecords = TotalRecords + teststr

        # Remove the number of files from the total Records.  This is because the program is counting the Header in the txt file.
        # if there are 7 files then we need to subtract 7 headers or the number in FileCounter variable.
        TotalRecords1 = TotalRecords - FileCounter‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Count Rows being processed with INSERT Cursor

BlakeTerhune — Fri, 07 Oct 2016 20:50:37 GMT

You can start the enumeration at 1 so you don't have to add one when you return i.

Pyhton enumerate()

python - How to enumerate a range of numbers starting at 1 - Stack Overflow

for i, l in enumerate(f, 1):‍

Re: Count Rows being processed with INSERT Cursor

JoshuaBixby — Mon, 10 Oct 2016 13:59:38 GMT

I suggest reading How to get line count cheaply in Python? if you haven't already. It provides high-performing options for counting file length as well as simple, more Pythonic ways of doing so.

Re: Count Rows being processed with INSERT Cursor

jaykapalczynski — Tue, 11 Oct 2016 12:59:04 GMT

your example shows how I am exactly doing it.....thanks

Re: Count Rows being processed with INSERT Cursor

ClintonDow1 — Tue, 11 Oct 2016 21:43:51 GMT

Since you are ultimately using an insert cursor, it may be simpler to have several functions: One function to read the lines out of the text file into a list and another function to loop through that list to insert to the feature class. Then as an intermediate step you can use Python's len() keyword to easily get the length of the list.

Re: Count Rows being processed with INSERT Cursor

DanPatterson_Retired — Wed, 12 Oct 2016 00:28:50 GMT

Joshua's answer was the correct one as you confirm... I have changed the correct answer to reflect this for future readers.

Re: Count Rows being processed with INSERT Cursor

ClintonDow1 — Wed, 12 Oct 2016 16:58:56 GMT

Just a small nit to pick, but the actual question being asked isn't answered by his response and could be misleading to novice programmers. It asks how to count rows being processed with an insert cursor, not how to count the number of rows in a text file. These may not be equivalent if the rows are inserted conditionally.