AnsweredAssumed Answered

How can I add values to this  Python dictionary more efficiently?

Question asked by cbevilacqua on Feb 8, 2019
Latest reply on Feb 11, 2019 by cbevilacqua

I've been building on some principles from Richard Fairhurst's BlogTurbo Charging Data Manipulation with Python Cursors and Dictionaries My goal is to map the average daily well volume for the past 90 days.  I have a table of daily volumes for about 1,300 wells.  There are several years worth of records in the table.  I also have a Python dictionary.  The keys in the dictionary are the unique identifier for each well represented in the table.  The values in the dictionary start as empty lists.  The following block of code populates the dictionary values with the daily volume values from the past 92 days for each well.  The code works, but it takes approximately 40 seconds for each well.  How can I modify the code to improve performance?  Please see the code and comments below.  Any suggestions would be greatly appreciated.  

 

i = 0 # This is a counter for testing purposes
table = acDaily # This is a SQL Server database table of daily water volumes.  There are ~1300 wells represented in the table, each with daily records covering several years.
fields = ['PROPNUM','D_DATE','WATER'] # PROPNUM is the unique identifier for each well in acDaily, D_DATE is the date of the daily volume, and WATER is the volume of produced water.

for prop in propNumDict: # Iterate through each item in propNumDict.  propNumDict is a Python dictionary with keys that are the unique values in the acDaily PROPNUM field.  The dictionary values start as empty lists.
    i += 1
    print str(i) + " - " + strftime("%Y-%m-%d %H:%M:%S")  # This is used to time the duration of each iteration, which is currently ~40 seconds.
    if i > 2:
        break
    else:
        with arcpy.da.SearchCursor(table, fields) as cursor: # Use a search cursor to go through the records in acDaily and if the PROPNUM matches the key in propNumDict and the records are from the past 92 days, add the water volume value to the dictionary.
            for row in cursor:
                if row[0] == prop and row[1] > tMinus92: # tMinus92 is a variable defined as datetime.datetime.today() - timedelta(days = 92).  It takes two days for the data to make it into this table.
                    propNumDict[prop].append(row[2])

print propNumDict
print strftime("%Y-%m-%d %H:%M:%S")

 

After the propNumDict dictionary is populated, I have another block of code that creates a new dictionary and populates it with the average values for each key in propNumDict.  Here's what that looks like:

avgDailyDict = {}
for key, values in propNumDict.iteritems():
    if float(len(values)) > 0:
        avgDailyDict[key] = sum(values)/float(len(values))

print avgDailyDict

 

arc gis python

dictionary

arcpy.da.searchcursor

Outcomes