list comprehension to evaluate each element of a list

clt_cabq

In my project I am reading a .csv which has two date values, these are at positions 5 and 14 in the row. The date values can either be null (indicated as '' in the .csv) or they can hold an actual date in the format of mm/dd/yyyy. In order to insert these rows into my geodatabase, I have to convert the '' value to a null, otherwise the value is not recognized as valid for a date field.

I can get the following code to work for each field:

empty = line[5]
if empty == '';
  line[5] = None
else:
  line[5]=empty

However, I was thinking I could implement a list comprehension to accomplish this more efficiently by using the following:

datelist = [line[5],line[14]]
datelist = [None if date == '' else date for date in datelist]

This doesn't seem to work, I get an error indicating a type mismatch with a date field.

Any thoughts on what I am misunderstanding or how I can approach this. Since I have a working solution its not critical so I am just trying to improve my skills and knowledge a bit here. Though since I am iterating over 20K records, I suppose every bit of efficiency helps.  

Hayden Welch

Is your null value two single quotes or an empty string?

date1,date2
'',dd/mm/yyy

-- OR --
date1,date2
,dd/mm/yyy

The former will evaluate to "''" when read, so your check is incorrect:

>>> # version 1
>>> print(row[4])
''
>>> repr(row[4])
'"\'\'"'
>>> # version 2
>>> print(row[4])

>>> repr(row[4])
"''"

 

To fix it that way, just modify your condition to check for the literal '' string:

datelist = [line[5],line[14]]
datelist = [None if date == "''" else date for date in datelist]

 

You can also do some fun stuff with the csv.DictReader if you want to avoid indexing:

from collections.abc import Iterator
from typing import Any
from pathlib import Path
from csv import DictReader

def format(line: str, *, sep: str=',') -> list[str]:
    return line.strip().split(sep)

def get_dates(fl: Path) -> Iterator[tuple[str | None, str | None]]:
    with fl.open('rt') as data:
        headers = format(data.readline())
        for line in data:
            row = dict(zip(headers, format(line)))
            yield (
                None if (dt:=row.get('date1')) == "''" else dt, 
                None if (dt:=row.get('date2')) == "''" else dt,
            )

def format_csv(fl: Path) -> Iterator[dict[str, Any]]:
    with fl.open('rt') as data:
        headers = format(data.readline())
        for row in DictReader(data, headers):
            row['date1'] = None if (dt:=row.get('date1')) == "''" else dt
            row['date2'] = None if (dt:=row.get('date1')) == "''" else dt
            yield row

 

This creates a generator for the CSV rows, which means you are only loading in one row at a time, which lets you filter them before adding them to a memory list. If you have a ton of data this can prevent you from loading all of it into memory every time you want to check for something:

# Initialize the generator (nothing is loaded in)            
reader = format_csv(Path('data.csv'))

# Create a list that contains only the rows of the
# csv where both date fields are populated
valid_rows = [
    row for row in reader 
    if row['date1'] and row['date2']
]

 

Additional Note on Generators:

Once a generator is exhausted (it's been iterated with a comprehension/for loop), It can't be iterated again:

>>> x = (i for i in range(10)) # generator comprehension syntax
>>> list(x)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> list(x)
[]
>>>