Latest Contributions by LauraBlackburn

Here's what I tried before... seqField = "Sequence" where_clause = arcpy.AddFieldDelimiters("clipLayer", seqField)+ sN arcpy.SelectLayerByAttribute_management("clipLayer", "NEW_SELECTION", where_clause)                # Clip roads w/ selected polygon - for this I will need a search cursor clipRoadsShp = arcpy.Clip_analysis(Road, "clipLayer", "in_memory")                # Use geometry/length to get the total length of clipped roads(in meters) print "calculating road lengths...." print 'Clip complete '  + strftime("%Y-%m-%d %H:%M:%S") g = arcpy.Geometry() geometryList = arcpy.CopyFeatures_management(clipRoadsShp, g) length = 0      for geometry in geometryList:              length +=geometry.length I think that the where-clause and the clipping really slowed the process down and my total processing time was going to be weeks!  Also, my polygons overlap and while I do have the advanced license, I am not sure the tabulate intersect tool would work because of the overlapping.  I will try the intersect tool and see if that helps.  I was thinking that creating a bunch of clip or intersect layers was a waste of space & time because I really only need them to compute the length of roads inside the polygon and that's it. ... View more

I am a bit unclear on how to write the syntax for performing the intersect method. I used your suggestion, Richard, with a dictionary to store the geometries of each record in the envelope layer.  However, line 7 is where I get confused (I added this and I know it's not written right).  I am sure at some point I need to tell it to intersect the road layer and output lines, in order to pass the length to the dictionary.  Then I need to pass the length of those lines that are intersecting the envelope back to the CSV file, via dataList.append(length).  I think line 9 calculates the length and passes it back to the dictionary, though I don't understand what envelopeDict[relateKey] does.  Is this just appending it to the dictionary or is it replacing the relateKey?  At the moment, I am thinking this snippet of code should follow all the other dictionaries and be just above the line where the dataList is written to the CSV file.                 envelopeFieldsList = ["Sequence", "Shape@"]                 envelopeDict = {}                 with arcpy.da.SearchCursor("clipLayer", envelopeFieldsList) as envelopes:                     for envelope in envelopes:                         relateKey = envelope[0]                         geometry = envelope[1]                         geometry.intersect(Road, 2)                         if relateKey in envelopeDict:                             envelopeDict[relateKey] += geometry.length                         else:                             envelopeDict[relateKey] = geometry.length ... View more

Hi Richard, I know it's a crazy analysis.  I am data mining in order to build a model that will help lessen surveillance trapping costs of the gypsy moth.  I used Network analyst to create a number of routes, at multiple distances between traps, in various locations in the US.  Now I am going into the routing data and extracting information on the time it takes to visit each trap location.  So, I am populating a CSV table that has the following data: block, scale (distance between traps), sequence (order of trap placement along route), time from previous trap, distance from previous trap, x, y, total time, total distance and road length. The envelope layers have all of the data from the trap placement/routing.  So, from those layers I am grabbing the time and distance from the previous trap  for sequence N and adding that value to the time and distance from the previous trap for sequence N+1.  So yes, I am using both N and N+1 data and combining them to eventually get the average time and distance for each and every trap location.  When the sequence is the last one along a route, I need to grab the value from the depot visits layer (this value doesn't exist in the envelope layer). Also for each trap location, I want to know the average road density in the local area and at the regional scale.  My envelopes were created at two scales, one is for computing the local area's road density and the other is for calculating the regional road density.  This is why I was doing the clipping.  I am currently looking into your intersect method and agree that this might be a quicker way to go.  And yes, I tend to default to shapefiles I guess that dates me, as I learned GIS at the ArcView 3.2 version.  I will try to stop doing that too. ... View more

Thanks Richard for all of your time and for walking me through all these new (to me) techniques. I attached a jpg that will hopefully show you what I am trying to accomplish.  The top image is one of my "envelope" layers with one record (aka. sequence id) shown in purple out of 144 records. The bottom image shows the envelope layer with the road layer that is being clipped.  What I want to return for each sequence is the length of the roads that are inside that individual polygon.  This is why I used a clip and not simply a selection.  When I perform a selection (which is done in the bottom image), I do not get a complete record of the roads that fall inside the selected polygon.  Some roads are left out, as shown below.  What I am ultimately trying to determine is the road density inside each polygon and to compute that I need to know the total length of roads that are inside of the polygon.  Let me know if you have any ideas on getting at this data in a different manner, as the clipping and selecting both seem to take up lots of time. ... View more

clipping one record at a time adds two minutes per loop!  And calculating the geometry only takes a second.  So, if there's a better way to perform the clip that would save me a ton of time! ... View more

# -*- coding: utf-8 -*- # --------------------------------------------------------------------------- # 2_FastTrapStatsAndFloatingRoadDensity.py # Altered on: 2015-6-8 10:00 am # Created by: L.M.Blackburn # ArcGIS 10.1 # Notes: Buffers and envelopes were created in first run-through of this code. #        Now, I calculate length for the clipped roads by adding #        a field and calculating geometry. Run statistics to get the sum length #        of roads in each square. export sum length, block name, centroid of #        square out to a text file.  I'm making edits on this version to run #        the code on Citrix and to speed up the entire process using dictionary #        keys instead of nested cursors. #        (2nd script for 4th part of analysis) # Needed data: Order sublayer from VRP #              List of fields to drop #              Roads network to get length data               # --------------------------------------------------------------------------- # Import arcpy module import arcpy, os, sys, traceback, csv from time import strftime # Set the necessary product code import arceditor import arcinfo # Set workspace environment - this is where the solved route layers are located arcpy.env.workspace = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15\\Orders\\' arcpy.env.overwriteOutput = True # --------------------Set local variables ------------------ # ---workspace for order layers ordersLyr = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15\\Orders\\' # ---workspace for buffer layers bufferLyr = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15\\Buffers\\' # ---set default buffer distance bufDefault = "6000 Meters" # ---workspace for envelope layers envelopeLyr = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15\\Envelopes\\' # ---workspace for route layers routeWkspc = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15\\Routes\\' # ---set workspace path for clipped roads outDataPath = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15\\ClipRoad\\' # ---set road layer - for use in clip & length Road = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\RoadsForSnap\\SDC Edge Source.shp' # ---fields to delete from Orders layers dropFields = ["Descriptio", "ServiceTim", "TimeWind_1", "TimeWind_2", "TimeWind_3", "TimeWindow", "MaxViolati", "MaxViola_1", \               "PickupQuan", "DeliveryQu", "Revenue", "SpecialtyN", "Assignment", "RouteName", "ViolatedCo", "CumulTrave", \               "CumulDista", "CumulTime", "ArriveCurb", "DepartCurb", "DepartTime", "WaitTime", "ViolationT", "CumulWaitT", \               "CumulViola", "ArriveTime", "SourceID", "SourceOID", "PosAlong", "SideOfEdge", "CurbApproa", "Status"] # ---fields for CSV file fieldNames = ['Block', 'Scale', 'BufferType', 'Sequence', 'FromPrevTr', 'FromPrevDi', 'x', 'y', 'bufferDist','totTime', \               'totDist', 'totalRdLength', '\n'] # ---List of fields from envelope layer from which to extract data sourceFieldsList = ["Sequence", "FromPrevTr", "FromPrevDi", "POINT_X", "POINT_Y", "BUFF_DIST"]               try:     print 'Script started at: ' + strftime("%Y-%m-%d %H:%M:%S")     print 'Opening CSV file...'     # Step 1 - Create table for output data     # Needed fields: From script [Block, Scale, Sequence, FromPrevTr, FromPrevDi, totTime, totDist, x, y, totalRdLength]     with open('F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15\\AllTrapsData_June9_new.csv', "wb") as f:         w = csv.writer(f, delimiter=',', lineterminator='\n')         w.writerow(fieldNames)         # Calculate totalTime and totalDistance using insert cursor         # loop through each record in the table - calculate values and...         # use selected features to clip roads layer & calculate geometry         # add values to the CSV table         print 'Populating CSV file...'                 envelopeList = []         for dirpath, dirnames, filenames in arcpy.da.Walk(envelopeLyr, datatype = 'FeatureClass', type = 'Polygon'):             for filename in filenames:                 envelopeList.append(os.path.join(filename))         for eLayer in envelopeList:             eLayerName = eLayer.split('_')             blockEID = eLayerName[1]             scaleEID = eLayerName[2].rstrip('.shp')             bufEID = eLayerName[0]             #print 'eLayer: ' + eLayer + ' Block: ' + blockEID + ' Scale: ' + scaleEID + ' BufferType: ' + bufEID             eLyrPath = envelopeLyr + eLayer                         #Make a layer from the feature class - needed for selecting records to be used in the clipping             arcpy.MakeFeatureLayer_management(eLyrPath, "clipLayer")             #-----------------------pasted code below----------------------             sourceFC = None              #select corresponding route layer file and grab sublayer: DepotVisits               print 'extracting time & distance to points. eLayer: ' + eLayer + ' Block: ' + blockEID + ' Scale: ' + scaleEID + ' BufferType: ' + bufEID               routeLyr = arcpy.mapping.Layer(routeWkspc + "TrapRoute_" + blockEID + "_" + scaleEID + ".lyr")               if routeLyr.isGroupLayer:                   for sublyrs in routeLyr:                       # print sublyrs.name                       if sublyrs.name == 'Depot Visits':                           arcpy.MakeFeatureLayer_management(sublyrs, "depotVisits")                          sourceFC = "depotVisits"               qLast = "= 2"             expression2 = arcpy.AddFieldDelimiters("depotVisits", "VisitType")+qLast             depotFieldsList = ["VisitType", "FromPrevTravelTime", "FromPrevDistance"]             depotVisitsDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, depotFieldsList, expression2)}                   sourceFC = eLyrPath                                          # Use list comprehension to build a dictionary from a da SearchCursor                sequenceDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, sourceFieldsList)}                    for key in sequenceDict:                   dataList = []                    dataList.append(str(blockEID))                    dataList.append(str(scaleEID))                    dataList.append(str(bufEID))                    dataList.append(str(key))                    dataList.append(str(list(sequenceDict[key])[0]))                                    dataList.append(str(list(sequenceDict[key])[1]))                    dataList.append(str(list(sequenceDict[key])[2]))                    dataList.append(str(list(sequenceDict[key])[3]))                    dataList.append(str(list(sequenceDict[key])[4]))                    n = int(key)                    sN = "=" + str(n)                    n2 = n +1                    sel = "=" + str(n2)                    prevTime = float(list(sequenceDict[key])[0])                    prevDist = float(list(sequenceDict[key])[1])                    #print prevTime                 #print prevDist                 postTime = ""                    # use a nested search to get values for total time and total distance                 # this loop is not running - I am not getting these values written to                 # the CSV table                 if n2 in sequenceDict:                      #values = list(sequenceDict[n2])                      #for row2 in values:                     postTime = float(sequenceDict[n2][0])                     totTime = prevTime + postTime                      dataList.append(str(totTime))                        postDist = float(sequenceDict[n2][1])                       totDist = prevDist + float(postDist)                        dataList.append(str(totDist))                                # if above for loop yeilds no results, calculate postTime and postDist differently                        # grab DepotVisits layer - if VisitType = "End" then grab values for "FromPrevTravelTime" & "FromPrevDistance"                        if postTime == "":                                                      #select corresponding route layer file and grab sublayer: DepotVisits                            print "extracting time & distance to Depot"                            if 2 in depotVisitsDict:                              row3 = list(depotVisitsDict[2])                              postTime2 = row3[1]                                totTime2 = prevTime + float(postTime2)                                print ("Depot Visit Time: {0}, {1}, {2}".format(prevTime, postTime2, totTime2))                                dataList.append(str(totTime2))                                postDist2 = row3[2]                                totDist2 = prevDist + float(postDist2)                                # print ("Depot Visit Distance: {0}, {1}, {2}".format(prevDist, postDist2, totDist2))                                dataList.append(str(totDist2))                            # ----------------------pasted code above----------------------                 print "Sequence: " + sN + ", Block: " + blockEID + ", Scale: " + scaleEID                 # select single record to use when clipping - this selection must be on a layer not fc                 # always create where clause using AddFieldDelimiters                 seqField = "Sequence"                 where_clause = arcpy.AddFieldDelimiters("clipLayer", seqField)+ sN                 arcpy.SelectLayerByAttribute_management("clipLayer", "NEW_SELECTION", where_clause)                                 # Clip roads w/ selected polygon - for this I will need a search cursor                 #clipRoads = outDataPath + 'rdsClip_' + blockEID + '_' + scaleEID + '_' + bufEID + '_' + str(n)                 #arcpy.Clip_analysis(Road, "clipLayer", clipRoads)                 #clipRoadsShp = clipRoads + ".shp"                        clipRoadsShp = arcpy.Clip_analysis(Road, "clipLayer", "in_memory")                                 # Use geometry/length to get the total length of clipped roads(in meters)                 print "calculating road lengths...."                 g = arcpy.Geometry()                 geometryList = arcpy.CopyFeatures_management(clipRoadsShp, g)                 length = 0                 for geometry in geometryList:                     length +=geometry.length                 # append length (meters) at end of line to csv dataList                 dataList.append(str(length))                 # dataList.append('\r\n')                 #print length                 arcpy.Delete_management(clipRoadsShp)                                 # Write dataList to csv file ---- may need to dedent this one more time ---                 w.writerow(dataList)                 # print dataList             print 'Script completed at: ' + strftime("%Y-%m-%d %H:%M:%S")     f.close() except:     f.close()     print 'Program failed at: ' + strftime("%Y-%m-%d %H:%M:%S")     print 'Check python errors.'             tb = sys.exc_info()[2]     tbinfo = traceback.format_tb(tb)[0]     pymsg = "PYTHON ERRORS:\nTraceback Info:\n" + tbinfo + "\nError Info:\n     " +        str(sys.exc_type) + ": " + str(sys.exc_value) + "\n"     msgs = "ARCPY ERRORS:\n" + arcpy.GetMessages(2) + "\n"         arcpy.AddError(msgs)     arcpy.AddError(pymsg)     print msgs     print pymsg         arcpy.AddMessage(arcpy.GetMessages(1))     print arcpy.GetMessages(1)     # print ("Points to select: {0}, Radius: {1}".format(centroidPath, radius)) Thanks so much, Richard, for all the time you are devoting to my problem!   The above code is currently working, though it's still slow. I have 360 polygon layers in the envelopeList that I am using to clip a roads layer.  Each layer in the envelopeList has multiple polygons  and I am calculating the length of roads that fall inside each polygon (which corresponds to the sequence field ID) to add to the CSV file.  I suppose this could be done for all of the polygons in a layer instead of one polygon at a time, then join the data back using the sequence id and a dictionary? My original code was created by me in whatever manner I could make it work.  I do not have formal programming experience and I am a wildlife biologist by training, so I have learned python on my own as I need to mine data for work.  I am sure that my code is not elegant, but I am learning a ton!  I know there are always many ways of getting at the same end result, and I simply use whatever methods I understand.  So, if there's a better way, I am all ears!  I would much rather have this process through my files in days instead of weeks!  So any time saving tips are greatly appreciated. ... View more

okay, I have all of the dictionaries working and my CSV file is being populated correctly.  I altered the road clipping to use "in_memory" processing too.  But, I still seem to be dealing with slow processing speeds.  It seems like it will take many days to complete. ... View more

Thanks again Richard! The sequence field is a short integer.  So, by changing it from a string to a number we are now going into that loop, however, I get a queue.empty error at line 136 postTime=row2[1].  Could you describe what we are doing in line 134? There should only be one record where n2 is in the sequenceDict.  What I think line 134 does, is put all of the records for n2 into a list.  Then postTime would be the second field in the sequenceDict where the first field = n2. Also, the speed on this script still seems to be slow.  I am pretty sure this is due to the clipping of roads and calculating the road length.  Is this something that could be done in-memory to help with the speed?  Would the syntax be something like below to replace lines 173-175: clipRoadsShp = arcpy.Clip_analysis(Road, "clipLayer", "in_memory") ... View more

# -*- coding: utf-8 -*- # --------------------------------------------------------------------------- # 2_FastTrapStatsAndFloatingRoadDensity.py # Altered on: 2015-6-8 10:00 am # Created by: L.M.Blackburn # ArcGIS 10.1 # Notes: Buffers and envelopes were created in first run-through of this code. #        Now, I calculate length for the clipped roads by adding #        a field and calculating geometry. Run statistics to get the sum length #        of roads in each square. export sum length, block name, centroid of #        square out to a text file.  I'm making edits on this version to run #        the code on Citrix and to speed up the entire process using dictionary #        keys instead of nested cursors. #        (2nd script for 4th part of analysis) # Needed data: Order sublayer from VRP #              List of fields to drop #              Roads network to get length data               # --------------------------------------------------------------------------- # Import arcpy module import arcpy, os, sys, traceback, csv from time import strftime # Set the necessary product code import arceditor import arcinfo # Set workspace environment - this is where the solved route layers are located arcpy.env.workspace = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15cont\\Orders\\' arcpy.env.overwriteOutput = True # --------------------Set local variables ------------------ # ---workspace for order layers ordersLyr = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15cont\\Orders\\' # ---workspace for buffer layers bufferLyr = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15cont\\Buffers\\' # ---set default buffer distance bufDefault = "6000 Meters" # ---workspace for envelope layers envelopeLyr = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15cont\\Envelope\\' # ---workspace for route layers routeWkspc = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15cont\\Routes\\' # ---set workspace path for clipped roads outDataPath = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15cont\\ClipRoad\\' # ---set road layer - for use in clip & length Road = 'F:\\Workspace\\Sandy\\GM_costAnalysis\\RoadsForSnap\\SDC Edge Source.shp' # ---fields to delete from Orders layers dropFields = ["Descriptio", "ServiceTim", "TimeWind_1", "TimeWind_2", "TimeWind_3", "TimeWindow", "MaxViolati", "MaxViola_1", \               "PickupQuan", "DeliveryQu", "Revenue", "SpecialtyN", "Assignment", "RouteName", "ViolatedCo", "CumulTrave", \               "CumulDista", "CumulTime", "ArriveCurb", "DepartCurb", "DepartTime", "WaitTime", "ViolationT", "CumulWaitT", \               "CumulViola", "ArriveTime", "SourceID", "SourceOID", "PosAlong", "SideOfEdge", "CurbApproa", "Status"] # ---fields for CSV file fieldNames = ['Block', 'Scale', 'BufferType', 'Sequence', 'FromPrevTr', 'FromPrevDi', 'x', 'y', 'bufferDist','totTime', \               'totDist', 'totalRdLength', '\n'] # ---List of fields from envelope layer from which to extract data sourceFieldsList = ["Sequence", "FromPrevTr", "FromPrevDi", "POINT_X", "POINT_Y", "BUFF_DIST"]               try:     print 'Script started at: ' + strftime("%Y-%m-%d %H:%M:%S")     print 'Opening CSV file...'     # Step 1 - Create table for output data     # Needed fields: From script [Block, Scale, Sequence, FromPrevTr, FromPrevDi, totTime, totDist, x, y, totalRdLength]     with open('F:\\Workspace\\Sandy\\GM_costAnalysis\\analysisMay15cont\\AllTrapsData_June8_test.csv', "wb") as f:         w = csv.writer(f, delimiter=',', lineterminator='\n')         w.writerow(fieldNames)         # Calculate totalTime and totalDistance using insert cursor         # loop through each record in the table - calculate values and...         # use selected features to clip roads layer & calculate geometry         # add values to the CSV table         print 'Populating CSV file...'                 envelopeList = []         for dirpath, dirnames, filenames in arcpy.da.Walk(envelopeLyr, datatype = 'FeatureClass', type = 'Polygon'):             for filename in filenames:                 envelopeList.append(os.path.join(filename))         for eLayer in envelopeList:             eLayerName = eLayer.split('_')             blockEID = eLayerName[1]             scaleEID = eLayerName[2].rstrip('.shp')             bufEID = eLayerName[0]             #print 'eLayer: ' + eLayer + ' Block: ' + blockEID + ' Scale: ' + scaleEID + ' BufferType: ' + bufEID             eLyrPath = envelopeLyr + eLayer                         #Make a layer from the feature class - needed for selecting records to be used in the clipping             arcpy.MakeFeatureLayer_management(eLyrPath, "clipLayer")             #-----------------------pasted code below----------------------             sourceFC = None              #select corresponding route layer file and grab sublayer: DepotVisits               print 'extracting time & distance to points. eLayer: ' + eLayer + ' Block: ' + blockEID + ' Scale: ' + scaleEID + ' BufferType: ' + bufEID               routeLyr = arcpy.mapping.Layer(routeWkspc + "TrapRoute_" + blockEID + "_" + scaleEID + ".lyr")               if routeLyr.isGroupLayer:                   for sublyrs in routeLyr:                       # print sublyrs.name                       if sublyrs.name == 'Depot Visits':                           arcpy.MakeFeatureLayer_management(sublyrs, "depotVisits")                          sourceFC = "depotVisits"               qLast = "= 2"             expression2 = arcpy.AddFieldDelimiters("depotVisits", "VisitType")+qLast             depotFieldsList = ["VisitType", "FromPrevTravelTime", "FromPrevDistance"]             depotVisitsDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, depotFieldsList, expression2)}                   sourceFC = eLyrPath                                          # Use list comprehension to build a dictionary from a da SearchCursor                sequenceDict = {r[0]:(r[1:]) for r in arcpy.da.SearchCursor(sourceFC, sourceFieldsList)}                    for key in sequenceDict:                   dataList = []                    dataList.append(str(blockEID))                    dataList.append(str(scaleEID))                    dataList.append(str(bufEID))                    dataList.append(str(key))                    dataList.append(str(list(sequenceDict[key])[0]))                                    dataList.append(str(list(sequenceDict[key])[1]))                    dataList.append(str(list(sequenceDict[key])[2]))                    dataList.append(str(list(sequenceDict[key])[3]))                    dataList.append(str(list(sequenceDict[key])[4]))                    n = int(key)                    sN = "=" + str(n)                    n2 = n +1                    sel = "=" + str(n2)                    prevTime = float(list(sequenceDict[key])[1])                    prevDist = float(list(sequenceDict[key])[2])                    # print sN                    postTime = ""                    # use a nested search to get values for total time and total distance                 # this loop is not running - I am not getting these values written to                 # the CSV table                 if str(n2) in sequenceDict:                      values = list(sequenceDict[str(n2)])                      for row2 in values:                           postTime = row2[1]                                                totTime = prevTime + float(postTime)                            print totTime                            dataList.append(str(totTime))                            postDist = row2[2]                            # print postDist                            totDist = prevDist + float(postDist)                            # print totDist                            dataList.append(str(totDist))                                # if above for loop yeilds no results, calculate postTime and postDist differently                        # grab DepotVisits layer - if VisitType = "End" then grab values for "FromPrevTravelTime" & "FromPrevDistance"                        if postTime == "":                                                      #select corresponding route layer file and grab sublayer: DepotVisits                            print "extracting time & distance to Depot"                            if 2 in depotVisitsDict:                              row3 = list(depotVisitsDict[2])                              postTime2 = row3[1]                                totTime2 = prevTime + float(postTime2)                                print ("Depot Visit Time: {0}, {1}, {2}".format(prevTime, postTime2, totTime2))                                dataList.append(str(totTime2))                                postDist2 = row3[2]                                totDist2 = prevDist + float(postDist2)                                # print ("Depot Visit Distance: {0}, {1}, {2}".format(prevDist, postDist2, totDist2))                                dataList.append(str(totDist2))                            # ----------------------pasted code above----------------------                 print "Sequence: " + sN + ", Block: " + blockEID + ", Scale: " + scaleEID                 # select single record to use when clipping - this selection must be on a layer not fc                 # always create where clause using AddFieldDelimiters                 seqField = "Sequence"                 where_clause = arcpy.AddFieldDelimiters("clipLayer", seqField)+ sN                 arcpy.SelectLayerByAttribute_management("clipLayer", "NEW_SELECTION", where_clause)                                 # Clip roads w/ selected polygon - for this I will need a search cursor                 clipRoads = outDataPath + 'rdsClip_' + blockEID + '_' + scaleEID + '_' + bufEID + '_' + str(n)                 arcpy.Clip_analysis(Road, "clipLayer", clipRoads)                 clipRoadsShp = clipRoads + ".shp"                        # Use geometry/length to get the total length of clipped roads(in meters)                 print "calculating road lengths...."                 g = arcpy.Geometry()                 geometryList = arcpy.CopyFeatures_management(clipRoadsShp, g)                 length = 0                 for geometry in geometryList:                     length +=geometry.length                 # append length (meters) at end of line to csv dataList                 dataList.append(str(length))                 # dataList.append('\r\n')                 print length                                 # Write dataList to csv file ---- may need to dedent this one more time ---                 w.writerow(dataList)                 # print dataList             print 'Script completed at: ' + strftime("%Y-%m-%d %H:%M:%S")     f.close() except:     f.close()     print 'Program failed at: ' + strftime("%Y-%m-%d %H:%M:%S")     print 'Check python errors.'             tb = sys.exc_info()[2]     tbinfo = traceback.format_tb(tb)[0]     pymsg = "PYTHON ERRORS:\nTraceback Info:\n" + tbinfo + "\nError Info:\n     " +        str(sys.exc_type) + ": " + str(sys.exc_value) + "\n"     msgs = "ARCPY ERRORS:\n" + arcpy.GetMessages(2) + "\n"         arcpy.AddError(msgs)     arcpy.AddError(pymsg)     print msgs     print pymsg         arcpy.AddMessage(arcpy.GetMessages(1))     print arcpy.GetMessages(1)     # print ("Points to select: {0}, Radius: {1}".format(centroidPath, radius)) The majority of the code seems to be working.  However, I am not getting any values returned from the if statement on line 133.  Any ideas on what might be happening here?  What I am doing in that loop is finding the next point in the route (sequence +1) and grabbing the values for "FromPrevTravelTime" and "FromPrevDistance" from the same layer or the sequenceDict .  Should this if statement be changed to a for statement? I am also wondering if I should order the sequence dictionary?  Would this save on processing time? ... View more

Hi Richard, Thank you for walking me through the use of the dictionary.  Dictionaries of this type are completely new to me, so I have been referring to your blog "Turbo Charging Data Manipulation..." to try and understand what the code is doing.  Here's a little background on what I am doing.  I used Network analyst to create routes for me.  Now I am querying those routes to get data on the time and distance traveled from one point to the next.  This works great until we get to the very last point or location in the route, identified by the final sequence #.  For the final point in the route, I still need to know how long (time & dist) it takes to get to the depot (ie. the depot visit where visit type =2).  This is a one to one join - there are only two records in the depot visit table and I am only interested in the one where visit type =2.  Hopefully this helps to clarify what I am doing with expression2. So, I don't think I need a one to many dictionary for the depot visits. There are only two records here and I really only need data from the second record. Is it worth making a dictionary for one record? I have added your code Richard, and started debugging it, but I continue to get an error at line 24 in your code above. I seem to keep getting an error at      else:         depotVisitsDict[relatekey].append(depot[1:]) Error Info:      <class 'Queue.Empty'>: ... View more

I am getting a bit hung up on implementing the dictionaries, mainly because I am grabbing values from a feature class and putting them into a CSV file.  So, I don't think I can use an update cursor, instead I am using w.writerow with a list of records, right?  Please correct me if I am wrong.  Also, I am not just joining data from one table to another.  I am grabbing data one row at a time so that I can perform some data manipulations.  Basically, I have a point (n) with information about the time and distance from the previous point (stops along a route).  My code is adding the values for time and distance for point (n) to the time and distance for point (n+1).  So, I am querying the same feature class twice to get data for point(n) and adding that to point(n+1). Richard, any help or insight you can provide is greatly appreciated.  I am a self-taught coder and I feel that I know just enough to be dangerous.  If there are any books or courses that you found helpful, I would love to hear about it! ... View more

Thanks Richard for your input. It's so great to have such fast and insightful feedback from this forum! So what's my best bet for increasing processing speed of this script?  Do I just write to the in-memory workspace and leave the nested cursors?  Or, should I incorporate both the in-memory workspace and the dictionary keys?  Which of these changes will have the most significant impact? ... View more

Ah, I knew I could count on this forum to find a solution to my problem.  I'll look into both of these options in the morning.  I knew the nested cursors was messy but couldn't find a work around, so thank you Ian for leading me to Richard Fairhurst's information. Laura ... View more

This was all part of a network analysis.  I had created 30 blocks that are 12x12km and placed points inside those blocks at 6 different scales - so for the 6K scale there are 4 points, for the 4K scale there are 9 points, for the 3K scale there are 16 points, for the 2K scale there are 36, for the 1K there are 144 and for the 0.5K there are 576 points.  I have no idea how many line segments  - I am using a roads network for that and clipping at two scales around each point to determine the average road density around the point.  So the road density is computed around  the point in a 6000 meter radius and a radius equal to the point spacing (either 500 m, 1000m, 2000m, 3000m, 4000m or 6000m). Anyway, I can't quite grasp why it would be so slow this time w/ the same code I used before.  It doesn't really seem to be taxing my system much when I look in the task manager - CPU usage 17% and physical memory 37%. Thanks for the link Blake - I'll check into that. ... View more