POST
|
Thanks again, Richard. That works great. The cursor runs through all the rows and populates the dictionary in about 4 seconds!
... View more
02-11-2019
09:39 AM
|
0
|
0
|
4109
|
POST
|
02-11-2019
09:38 AM
|
0
|
0
|
4109
|
POST
|
Thanks for the feedback, Richard. This looks so obvious now. I had previously tried incorporating the date query as part of the search cursor, but hadn't been able to resolve SQL syntax errors. I still have my loops reversed though. I'll give your solution a try on Monday, hopefully, and will reply with results.
... View more
02-08-2019
03:24 PM
|
0
|
2
|
4109
|
POST
|
I've been building on some principles from Richard Fairhurst's Blog/blogs/richard_fairhurst/2014/11/08/turbo-charging-data-manipulation-with-python-cursors-and-dictionaries My goal is to map the average daily well volume for the past 90 days. I have a table of daily volumes for about 1,300 wells. There are several years worth of records in the table. I also have a Python dictionary. The keys in the dictionary are the unique identifier for each well represented in the table. The values in the dictionary start as empty lists. The following block of code populates the dictionary values with the daily volume values from the past 92 days for each well. The code works, but it takes approximately 40 seconds for each well. How can I modify the code to improve performance? Please see the code and comments below. Any suggestions would be greatly appreciated. i = 0 # This is a counter for testing purposes
table = acDaily # This is a SQL Server database table of daily water volumes. There are ~1300 wells represented in the table, each with daily records covering several years.
fields = ['PROPNUM','D_DATE','WATER'] # PROPNUM is the unique identifier for each well in acDaily, D_DATE is the date of the daily volume, and WATER is the volume of produced water.
for prop in propNumDict: # Iterate through each item in propNumDict. propNumDict is a Python dictionary with keys that are the unique values in the acDaily PROPNUM field. The dictionary values start as empty lists.
i += 1
print str(i) + " - " + strftime("%Y-%m-%d %H:%M:%S") # This is used to time the duration of each iteration, which is currently ~40 seconds.
if i > 2:
break
else:
with arcpy.da.SearchCursor(table, fields) as cursor: # Use a search cursor to go through the records in acDaily and if the PROPNUM matches the key in propNumDict and the records are from the past 92 days, add the water volume value to the dictionary.
for row in cursor:
if row[0] == prop and row[1] > tMinus92: # tMinus92 is a variable defined as datetime.datetime.today() - timedelta(days = 92). It takes two days for the data to make it into this table.
propNumDict[prop].append(row[2])
print propNumDict
print strftime("%Y-%m-%d %H:%M:%S") After the propNumDict dictionary is populated, I have another block of code that creates a new dictionary and populates it with the average values for each key in propNumDict. Here's what that looks like: avgDailyDict = {}
for key, values in propNumDict.iteritems():
if float(len(values)) > 0:
avgDailyDict[key] = sum(values)/float(len(values))
print avgDailyDict arc gis python dictionary arcpy.da.searchcursor
... View more
02-08-2019
12:03 PM
|
0
|
7
|
4455
|
POST
|
Thanks for the reply @Dan Patterson. I'm looking at gas well production and several variables that influence it. It turns out that the engineers I'm supporting may view the data relationships as linear. I have a follow-up meeting with one of them next week to discuss. I'll post an update afterward.
... View more
01-26-2018
06:17 AM
|
0
|
0
|
1100
|
POST
|
I've been exploring the Spatial Statistics tools in ArcGIS 10.1, specifically the Modeling Spatial Relationships toolbox. From reading the help documentation, I've gathered that these tools are only appropriate for scenarios in which the relationships between explanatory and dependent variables are linear. For scenarios with nonlinear relationships, the help documentation suggests either transforming the data to make it linear, or using a nonlinear regression method. Does anyone have any advice about moving forward with either of these suggestions for working with nonlinear relationships? I don't know how I would go about transforming my data, and I haven't been able to find any nonlinear regression tools in ArcGIS.
... View more
01-25-2018
03:46 PM
|
1
|
2
|
1583
|
BLOG
|
I'm not sure if this is your problem or not, but you are missing a closing parenthesis at the end of this line: keyValue = updateRow[0]+ "," + str(updateRow[1]) + "," + str(updateRow[2]
... View more
03-22-2017
06:58 AM
|
0
|
0
|
24583
|
POST
|
Thanks again Richard. Python and programming in general are new to me, so I always appreciate the basic debugging tips. I did some basic QC on my input feature classes and confirmed that there are no geometry errors. The dictionary appears to be correct. It contains 335 items, which is the number I expect based on the input I used to create sticksRF_split. Here is a sample of what the dictionary output looks like: u'J7MIJDJRDM': [(1368.551930715828, <Polyline object at 0x13beb890[0x13beb860]>)],
u'J7MIJLGT4M': [(1571.5487528800747, <Polyline object at 0x1346a510[0x1346a4e0]>)],
u'J7MIJDRLXQ': [(1347.2159353990594, <Polyline object at 0x13b53110[0x13b530e0]>)],
u'M5QMJ0CC2L': [(1313.688077024998, <Polyline object at 0x13bebf90[0x13bebf60]>)],
u'J7MIJ8HJZQ': [(1344.7776753085323, <Polyline object at 0x13beb650[0x13beb620]>)],
u'J7MIJ5QHMQ': [(1330.1471538449503, <Polyline object at 0x1346add0[0x1346ada0]>)],
u'K99ID3MOXV': [(1339.5960776588317, <Polyline object at 0x13471c70[0x13471c40]>)],
u'J7MIIT5TJQ': [(1344.1680296403613, <Polyline object at 0x2b989d0[0x2b98980]>)],
u'KA8ML48AOW': [(1385.925539247241, <Polyline object at 0x13471570[0x13471540]>)],
u'KA8MHNX69W': [(1461.8208431411147, <Polyline object at 0x1346a890[0x1346a860]>)],
u'J7MIJDCPVM': [(1151.2595366475489, <Polyline object at 0x134717b0[0x13471780]>)],
u'J7MIJKITUM': [(2126.2848170537795, <Polyline object at 0x1346ac10[0x1346abe0]>)],
u'L2OLSDE8NP': [(1409.090485039698, <Polyline object at 0x134714f0[0x134714c0]>)],
u'J7MIJB3R7Q': [(1368.551943220522, <Polyline object at 0x13beb850[0x13beb820]>)],
u'P69KSAA02B': [(1573.9873186602183, <Polyline object at 0x13466f30[0x13466f00]>)],
u'J7MIJASTCQ': [(1347.8256852251254, <Polyline object at 0x134664f0[0x134664c0]>)], I checked a few of the dictionary items in ArcMap and confirmed that the first value in the dictionary is the length of the longest line associated with each key. I took your advice @Anonymous User Fairhurst and tried some counts and prints. Since the dictionary looks good, I've been focusing on the update cursor, as shown below. I'm getting some odd results that I think are pointing me in the right direction. When I run the code below to the print statement line 3, I get a list of 1,011 PROPNUM values. If I run the entire block of code, I get 546 values from the print statement on line 3, and then I get the "IndexError: list index out of range" message. with arcpy.da.UpdateCursor(targetFC, ["PROPNUM_GIS","SHAPE@"]) as uCur:
for count, row in enumerate(uCur, start=1):
print "This is the count of rows in the update cursor: " + str(count) + " - PROPNUM = " + row[0] # This should match the number of sticks in targetFC (i.e., sticksRF)
# The count of rows in the update cursor = the number of records in targetFC (i.e., sticks RF; N = 1011) when I run the update cursor only
# to this print statement. When I run this entire block of code the interpreter stops enumerating at 546.
# print row
PROPNUM = row[0]
# print PROPNUM
if PROPNUM in valueDict:
print "This is the count of PROPNUM in valueDict: " + str(count)
row[1] = valueDict[PROPNUM][1]
# print row[1]
uCur.updateRow(row)
I think the problem I am having may be similar to one described on stackoverflow. In this case I am using a list of dictionary key values, and when I get to the end of the list, I get the error. Instead, I need to use an index. Am I on the right track here? The code below runs but neither modifies the feature geometry, nor gives messages in the Python interpreter. try:
with arcpy.da.UpdateCursor(targetFC, ["PROPNUM_GIS","SHAPE@"]) as uCur:
for row in uCur:
# print "This is the count of rows in the update cursor: " + str(count) + " - PROPNUM = " + row[0] # This should match the number of sticks in targetFC (i.e., sticksRF)
# The count of rows in the update cursor = the number of records in targetFC (i.e., sticks RF; N = 1011) when I run the update cursor only
# to this print statement. When I run this entire block of code the interpreter stops enumerating at 546.
# print row
PROPNUM = row[0]
# print PROPNUM
if PROPNUM in range(len(valueDict)):
# print "This is the count of PROPNUM in valueDict: " + str(count)
row[1] = [valueDict[PROPNUM] + valueDict[PROPNUM+1] for PROPNUM in range(len(valueDict)-1)]
print row[1]
uCur.updateRow(row)
except Exception as e:
arcpy.AddMessage(str(e.message))
... View more
02-16-2017
12:10 PM
|
0
|
1
|
504
|
POST
|
Thanks Richard. When the script gets to line 30 above, i.e., row[1] = valueDict[PROPNUM][1], I get the following message in the Python interpreter: Traceback (most recent call last):
File "\\qep\dnshare\GIS\Haynesville\Scripts\ShortenSticksWupdateCursor_vRichard.py", line 42, in <module>
row[1] = valueDict[PROPNUM][1]
IndexError: list index out of range If I understand the meaning of this error message, the update cursor is looking for a field that is not provided in the field list when the cursor was defined. However, SHAPE@ is included in the cursor definition as the second field, with an index number of 1 and referred to as row[1] on line 30. I don't think the message is referring to SHAPE@ in the dictionary since a dictionary is not a list. Thoughts?
... View more
02-15-2017
05:36 PM
|
0
|
3
|
504
|
POST
|
Thanks @Ian Murray. I'll print-tweak-reprint and let you know how it goes.
... View more
02-15-2017
04:46 PM
|
0
|
0
|
1213
|
POST
|
While I was troubleshooting the code that I had modified from what @ Richard Fairhurst provided, I realized that I can make this much simpler. This post on stackexchange made me realize that instead of creating a new feature class and populating it with an insert cursor, I should be able to use the dictionary I created with Richard's code to modify my original feature class (sticksRF) using an update cursor. The problem I have now has something to do with the geometry being provided by the SHAPE@ token. Here is my code: import arcpy
arcpy.env.overwriteOutput = True
sourceFC = r'\\qep\dnshare\GIS\Haynesville\WorkspaceGDBs\HaynesvilleRecovery.gdb\sticksRF_split'
targetFC = r'\\qep\dnshare\GIS\Haynesville\WorkspaceGDBs\HaynesvilleRecovery.gdb\sticksRF'
sourceFields = ["PROPNUM_GIS","Shape_Length","SHAPE@"]
print sourceFields
print "Using search cursor to create dictionary of features with greatest Shape_Length value for each unique value in PROPNUM_GIS field"
valueDict = {}
with arcpy.da.SearchCursor(sourceFC, sourceFields) as searchRows:
for searchRow in searchRows:
keyValue = searchRow[0] # PROPNUM_GIS is the key value searchRow?
if not keyValue in valueDict:
valueDict[keyValue] = [searchRow[1:]]
elif valueDict[keyValue][0] < searchRow[1]:
valueDict[keyValue] = [searchRow[1:]]
print valueDict # Use print to make sure dictionary is right. Should be a list of unique PROPNUMs and longest lengths. Looks good, 2/14/17.
with arcpy.da.UpdateCursor(targetFC, ["PROPNUM_GIS","Shape_Length","SHAPE@"]) as uCur:
for row in uCur:
PROPNUM = row[0]
if PROPNUM in valueDict:
row[2] = valueDict[PROPNUM]
uCur.updateRow(row) And here is the error I get: Traceback (most recent call last):
File "<editor selection>", line 6, in <module>
TypeError: cannot read geometry sequance, expecting list of floats Any suggestions?
... View more
02-15-2017
12:08 PM
|
0
|
7
|
1213
|
POST
|
import arcpy
arcpy.env.overwriteOutput = True
sourceFC = r'\\qep\dnshare\GIS\Haynesville\WorkspaceGDBs\HaynesvilleRecovery.gdb\sticksRF_split'
field_names = []
fields = arcpy.ListFields(sourceFC)
for field in fields:
if field.editable:
field_names.append(field.name)
all_editable_fields = field_names
sourceFieldsList = ["PROPNUM_GIS", "Shape_Length"] + all_editable_fields
print sourceFieldsList
valueDict = {}
with arcpy.da.SearchCursor(sourceFC, sourceFieldsList) as searchRows:
for searchRow in searchRows:
keyValue = searchRow[0] # PROPNUM_GIS is the key value
if not keyValue in valueDict:
valueDict[keyValue] = [searchRow[1:]]
elif valueDict[keyValue][0] < searchRow[1]:
valueDict[keyValue] = [searchRow[1:]]
print valueDict
arcpy.CreateFeatureclass_management(r'\\qep\dnshare\GIS\Haynesville\WorkspaceGDBs\HaynesvilleRecovery.gdb', "sticksTrimmed", "POLYLINE", sourceFC, "DISABLED", "DISABLED", sourceFC)
lines = arcpy.da.InsertCursor(r'\\qep\dnshare\GIS\Haynesville\WorkspaceGDBs\HaynesvilleRecovery.gdb\sticksTrimmed', all_editable_fields)
for propnum_gis in valueDict:
fieldValues = valueDict[propnum_gis][1:] Thanks Richard. I tried your code with some slight modifications, as shown above. The output feature class, sticksTrimmed, is empty. However, it does contain all of the necessary fields. I'm not really sure what is going on with the dictionary and if-elif statements, so I'm a bit stumped. One thing I did notice is that when I print valueDict, the key-value combinations are not really comparable. For example: {u'J7MIIN2UFQ': [(1368.5519332219944, (468190.93242110324, 3570500.798905503), u'J7MIIN2UFQ', u'1PDP', 1632.66115520795, None, 5584814.0, None, 4490.0, 1368.5520000000001)], ...} I'm going to try to simplify this a bit, since all I really need in the output is the geometry and PROPNUM values.
... View more
02-13-2017
03:16 PM
|
0
|
2
|
1213
|
POST
|
I'm using ArcGIS 10.1, Advanced license. I'm working with file geodatabase feature classes. I have a line feature class called sticksRF_2split. Each record in this feature class has a unique value in a field called PROPNUM_GIS. I run this feature class through the arcpy.SplitLineAtPoint_management tool. The output, sticksRF_split, has two records with the same PROPNUM_GIS value. From these pairs of matching PROPNUM_GIS values, I want to create a feature class that contains only the lines with the greater value in the SHAPE_Length field. I'm thinking the necessary approach would involve a dictionary with PROPNUM_GIS as the key and SHAPE_Length as the values, and then compare the values. I'm having a challenge finding any documentation about how to accomplish this. Is this the best way to approach this problem? Any advice or suggestions would be greatly appreciated.
... View more
02-13-2017
11:08 AM
|
0
|
13
|
2399
|
POST
|
Thanks Blake. That works, and I learned something new!
... View more
01-09-2017
10:49 AM
|
0
|
0
|
2257
|
Title | Kudos | Posted |
---|---|---|
1 | 01-25-2018 03:46 PM | |
1 | 12-06-2016 01:33 PM |
Online Status |
Offline
|
Date Last Visited |
11-11-2020
02:23 AM
|