Nested Loop - Site Comparison

[BenWeinstein]

Hi!

I'm new to python, but i believe i've exhausted my tools in modelbuilder. I have a cost path model that creates the best single path from every site, to every other site and then measures the cost path length. 64 sites, so a total of 2016 possible paths.

My sites are all in one table.

The model holds the first row as a source, and labels all others as destinations. So the first iteration the FID = 0 is the source and FID's 1-63 are the destinations. 2nd iteration FID = 1 is the source and FID's 2-63 are the destinations....

All i need is the model to iteratively (cursor?) go through and select each row from the "Possible destinations" (1,2,..,63) and perform the downstream tasks. I've called this a nested loop because i still need the entire model to iterate 63 times to grab the 63 sources and there corresponding possible destinations.

Below is my script, exported from modelbuilder

# Import system modules
import sys, string, os, arcgisscripting

# Create the Geoprocessor object
gp = arcgisscripting.create()

# Check out any necessary licenses
gp.CheckOutExtension("spatial")

# Load required toolboxes...
gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Spatial Analyst Tools.tbx")
gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Data Management Tools.tbx")
gp.AddToolbox("C:/Program Files (x86)/ET SpatialTechniques/ET GeoWizards 10.0 for ArcGIS 9.2 and 9.3/ET GeoWizards.tbx")
gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Analysis Tools.tbx")
gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Conversion Tools.tbx")

# Script arguments...
Cost_Path_Raster_from_Source_to_Destination_ = sys.argv[1]
if Cost_Path_Raster_from_Source_to_Destination_ == '#':
Cost_Path_Raster_from_Source_to_Destination_ = "H:\\BenModel\\CostPat_Jord1" # provide a default value if unspecified

v_FID_____n_ = sys.argv[2]
if v_FID_____n_ == '#':
v_FID_____n_ = "\"FID\" = %n%" # provide a default value if unspecified

Input_Sites = sys.argv[3]
if Input_Sites == '#':
Input_Sites = "Jordani1kmSites" # provide a default value if unspecified

# Local variables...
CostPath_Output = "H:\\BenModel\\PathoutputError.shp"
Vector_CostPath = "H:\\BenModel\\RasterT_CostPat1.shp"
v___ = "H:\\BenModel\\CostPat_Jord1"
v__ = "H:\\BenModel\\RasterT_CostPat1.shp"
Destination_Site = "H:\\BenModel\\Jordani1kmSites_Select1_Sele.shp"
Row_Count = "63"
= "H:\\BenModel\\Jordani1kmSites_Select1_Sele.shp"
Path_Output = "H:\\BenModel\\PathoutputError.shp"
Extent = "227845 3922854 315204.999984 3963250.127696"
Possible_Destinations_for_Selected_Site_shp = "H:\\BenModel\\Jordani1kmSites_Select1.shp"
Final_Output = "H:\\BenModel\\PathoutputError.shp"
Source = "Jordani1kmSites"
Cost_Distance_n_ = "C:\\Users\\Ari\\Desktop\\BackupModel\\test\\Cdist%n%"
Backlink_n_ = "C:\\Users\\Ari\\Desktop\\BackupModel\\test\\blink%n%"
Extent__2_ = "227845 3922854 315204.999984 3963250.127696"
Energy_Friction_Surface = "H:\\BenModel\\finaltran_eb"
Jordani1kmSites = "Jordani1kmSites"

# Process: Select Site to act as source...
gp.SelectLayerByAttribute_management(Input_Sites, "NEW_SELECTION", v_FID_n_)

# Process: Cost Distance...
tempEnvironment0 = gp.extent
gp.extent = "227845 3922854 315204.999984 3963250.127696"
gp.CostDistance_sa(Source, Energy_Friction_Surface, Cost_Distance_n_, "", Backlink_n_)
gp.extent = tempEnvironment0

# Process: Select Individual Site...
gp.Select_analysis(Jordani1kmSites, Possible_Destinations_for_Selected_Site_shp, "\"FID\" > %n%")

# Process: Get Count...
gp.GetCount_management(Possible_Destinations_for_Selected_Site_shp)

#HERE -> i need everything downstream of here to be iterated (a for loop based on the row count above) for each row in the above file!


# Process: Selects Row in Possible Destinations...
gp.Select_analysis(Possible_Destinations_for_Selected_Site_shp, Destination_Site, "\"FID\" = %n%")

# Process: Cost Path...
tempEnvironment0 = gp.extent
gp.extent = "227845 3922854 315204.999984 3963250.127696"
gp.CostPath_sa(Destination_Site, Cost_Distance_n_, Backlink_n_, Cost_Path_Raster_from_Source_to_Destination_, "BEST_SINGLE", "FID")
gp.extent = tempEnvironment0

# Process: Raster to Polyline...
gp.RasterToPolyline_conversion(Cost_Path_Raster_from_Source_to_Destination_, Vector_CostPath, "ZERO", "0", "NO_SIMPLIFY", "VALUE")

# Process: Delete...
gp.Delete_management(Cost_Path_Raster_from_Source_to_Destination_, "")

# Process: Append...
gp.Append_management("H:\\BenModel\\RasterT_CostPat1.shp", Path_Output, "NO_TEST", "ARCID 'ARCID' true false false 6 Long 0 6 ,First,#,H:\\BenModel\\RasterT_CostPat1.shp,ARCID,-1,-1;GRID_CODE 'GRID_CODE' true false false 6 Long 0 6 ,First,#,H:\\BenModel\\RasterT_CostPat1.shp,GRID_CODE,-1,-1;FROM_NODE 'FROM_NODE' true false false 6 Long 0 6 ,First,#;TO_NODE 'TO_NODE' true false false 6 Long 0 6 ,First,#;Length 'Length' true false false 13 Float 0 0 ,First,#;ID 'ID' true false false 4 Short 0 4 ,First,#", "")

# Process: Delete (2)...
gp.Delete_management(Vector_CostPath, "")

# Process: Delete (3)...
gp.Delete_management(Destination_Site, "")

# Process: Calculate Length...
gp.toolbox = "C:/Program Files (x86)/ET SpatialTechniques/ET GeoWizards 10.0 for ArcGIS 9.2 and 9.3/ET GeoWizards.tbx";
gp.ET_GPCalculateLength(Path_Output, "", "", "")

Thanks for any help

Ben Weinstein
University of Minnesota

[BenWeinstein]

Update!

I'm so close to finishing this model. Everything executes perfectly the first iteration, and sometimes the second. Its variable. However, at somepoint at the beginning of the source loop (labelled Red) i get the accursed cannot ger exclusive schema lock error: ExecuteError: ERROR 000464: Cannot get exclusive schema lock. Either being edited or in use by another application.

. Absolutely nothing else is open, there are other processes running. I've read in places that the searchcursors are having trouble with the schema lock repeats, since it creates a "virtual arcmap" run. Can anyone spy a reason that the first run through, and sometimes the second, works, but it fails as it dives back in?

# Import system modules
import sys, string, os, arcgisscripting

# Create the Geoprocessor object
gp = arcgisscripting.create()

# Check out any necessary licenses
gp.CheckOutExtension("spatial")

# Load required toolboxes...
gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Spatial Analyst Tools.tbx")
gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Data Management Tools.tbx")
gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Analysis Tools.tbx")
gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Conversion Tools.tbx")

# Local variables...
CostPath_Output = "C:\\Users\\Ari\\Desktop\\BackupModel\\test\\path%n%"
Vector_CostPath = "C:\\Users\\Ari\\Desktop\\BackupModel\\test\\Vpath%n%.shp"
Path_Output = "C:\Users\Ari\Desktop\BackupModel\PathOutput.shp"
Extent = "227845 3922854 315204.999984 3963250.127696"
Cost_Distance_n_ = "C:\\Users\\Ari\\Desktop\\BackupModel\\test\\Cdist%n%"
Backlink_n_ = "C:\\Users\\Ari\\Desktop\\BackupModel\\test\\blink%n%"
Extent__2_ = "227845 3922854 315204.999984 3963250.127696"
Energy_Friction_Surface = "H:\\BenModel\\finaltran_eb"
Jordani1kmSites = "C:\\Users\\Ari\\Desktop\\BackupModel\Jordani1kmSites.shp"
Input_Sites = "C:\\Users\\Ari\\Desktop\\BackupModel\\Jordani1kmSites.shp"
Destination_Site = "C:\\Users\\Ari\\Desktop\\BackupModel\\test\\Site_%n%_Variable.shp"
Possible_Destinations = "C:\\Users\\Ari\\Desktop\\BackupModel\\test\\Site_%n%.shp"

import arcgisscripting
gp = arcgisscripting.create(9.3)
# Import system modules
import sys, string, os, arcgisscripting

# Create the Geoprocessor object
gp = arcgisscripting.create()
gp.overwriteoutput = 1

# Load required toolboxes...
gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Analysis Tools.tbx")
# Open a searchcursor
import sys, string, os, arcgisscripting
sources = gp.searchcursor(Jordani1kmSites)

# Get the first feature in the searchcursor
#
source = sources.next()
currentstate = ""
# Iterate through the rows in the cursor
#
while source:

if currentstate != source.Site_Name:
currentstate = source.Site_Name
print "Site: %s," % (source.Site_Name)
exp = "FID" + " = " + str(source.getvalue("Fid"))
print exp
print "Selected Desired Row"
gp.MakeFeatureLayer(Input_Sites,"lyr")
# Process: Select Site to act as source...
gp.SelectLayerByAttribute_management("lyr", "NEW_SELECTION", exp)
gp.overwriteoutput = 1
# Process: Cost Distance...
tempEnvironment0 = gp.extent
gp.extent = "227845 3922854 315204.999984 3963250.127696"
gp.CostDistance_sa("lyr", Energy_Friction_Surface, Cost_Distance_n_, "", Backlink_n_)
gp.extent = tempEnvironment0
gp.overwriteoutput = 1

print "Cost Distance"
gp.overwriteoutput = 1
exp = "FID" + " > " + str(source.getvalue("Fid"))
# Process: Select Individual Site...
gp.Select_analysis(Jordani1kmSites, Possible_Destinations, exp)
print "possible destinations created"
import arcgisscripting
gp = arcgisscripting.create(9.3)
gp.overwriteoutput = 1
import arcgisscripting
gp = arcgisscripting.create(9.3)

# Open a searchcursor

rows = gp.searchcursor(Possible_Destinations)

# Get the first feature in the searchcursor
#
row = rows.next()
currentstate = ""
# Iterate through the rows in the cursor
#
while row:

if currentstate != row.Site_Name:
currentstate = row.Site_Name
print "Site: %s," % (row.Site_Name)
exp = "FID" + " = " + str(row.getvalue("Fid"))
print exp
# Process: Make Feature Layer...
gp.MakeFeatureLayer_management(Possible_Destinations, "Dest")
# Process: Select Layer by Attribute...
gp.SelectLayerbyAttribute_management("Dest","NEW_SELECTION",exp)
print "Selected Desired Row"
# Process: Cost Path...
tempEnvironment0 = gp.extent
gp.extent = "227845 3922854 315204.999984 3963250.127696"
gp.CostPath_sa("Dest", Cost_Distance_n_, Backlink_n_, CostPath_Output, "EACH_ZONE")
gp.extent = tempEnvironment0
gp.overwriteoutput = 1
print "Cost Path"
# Process: Raster to Polyline...
gp.RasterToPolyline_conversion(CostPath_Output, Vector_CostPath, "ZERO", "0", "NO_SIMPLIFY", "VALUE")
print "Polyline"
# Process: Delete...
gp.Delete_management(CostPath_Output, "")
print "Delete"
# Process: Append...
gp.Append_management(Vector_CostPath,Path_Output,"NO_TEST")
print "append"
# Process: Delete (2)...
gp.Delete_management(Vector_CostPath, "")
print "delete"
print "Row Completed"
row = rows.next()
gp.Delete_Management("lyr")
gp.Delete_Management("Dest")
print "inside loop complete"
source = sources.next()
print "Done"

[GerryGabrisch]

I am not sure if this is your problem but when you are finished with a cursor be sure to use something like

del row, rows

By not deleting the row and rows (or whatever you name them) you can get schema locks when you try to run another cursor on the same data.

[BenWeinstein]

I am not sure if this is your problem but when you are finished with a cursor be sure to use something like

del row, rows

By not deleting the row and rows (or whatever you name them) you can get schema locks when you try to run another cursor on the same data.

Absolutely correct. I figured it out shortly afterwards. My only remaining issue, is the python version gives me SLIGHTLY different lengths then when i do a single test path in ArcGIS. I randomly choose a source and destination site, and the path i get for the python and arcgis results are like 100m different. The differences are always different, but never more than 1 or 2%.

Something about projections? Ideas? Its consistent enough to weird me out, but i can probably move forward and just increase my error bars slightly.

[LukeBadgerow]

Ben --

My gut reaction is that you might want to check the way that python is rounding, but as it looks like you're letting the geoprocessor do all of your math for you I'd say that you should be getting basically what you would expect compared to the model builder version.

As a note though, it looks like you re-declare the gp and the property overwrite a number of times bot at the 9.3 and 9.2 levels.  From a consistency standpoint I would pick one, declare it at the top of the script and stick with it.  PVM will be smart enough to hold that as long as you're running the module.  So any call to arcgisscripting.create(9.3) after line # 2, and any call to gp.overwriteoutput after line # 8

only import once as python will just ignore you anyway, so any import statement after line #13.

So my thoughts, pull out the spurious and extra import and assignments to the gp, then give it another round of testing and see what happens from there

import arcgisscripting
gp = arcgisscripting.create(9.3)
# Import system modules
import sys, string, os, arcgisscripting

# Create the Geoprocessor object
gp = arcgisscripting.create()
gp.overwriteoutput = 1

# Load required toolboxes...
gp.AddToolbox("C:/Program Files (x86)/ArcGIS/ArcToolbox/Toolboxes/Analysis Tools.tbx")
# Open a searchcursor
import sys, string, os, arcgisscripting
sources = gp.searchcursor(Jordani1kmSites)

# Get the first feature in the searchcursor
#
source = sources.next()
currentstate = ""
# Iterate through the rows in the cursor
#
while source:

if currentstate != source.Site_Name:
currentstate = source.Site_Name
print "Site: %s," % (source.Site_Name)
exp = "FID" + " = " + str(source.getvalue("Fid"))
print exp
print "Selected Desired Row"
gp.MakeFeatureLayer(Input_Sites,"lyr")
# Process: Select Site to act as source...
gp.SelectLayerByAttribute_management("lyr", "NEW_SELECTION", exp)
gp.overwriteoutput = 1
# Process: Cost Distance...
tempEnvironment0 = gp.extent
gp.extent = "227845 3922854 315204.999984 3963250.127696"
gp.CostDistance_sa("lyr", Energy_Friction_Surface, Cost_Distance_n_, "", Backlink_n_)
gp.extent = tempEnvironment0
gp.overwriteoutput = 1

print "Cost Distance"
gp.overwriteoutput = 1
exp = "FID" + " > " + str(source.getvalue("Fid"))
# Process: Select Individual Site...
gp.Select_analysis(Jordani1kmSites, Possible_Destinations, exp)
print "possible destinations created"
import arcgisscripting
gp = arcgisscripting.create(9.3)
gp.overwriteoutput = 1
import arcgisscripting
gp = arcgisscripting.create(9.3)

# Open a searchcursor

rows = gp.searchcursor(Possible_Destinations)

# Get the first feature in the searchcursor
#
row = rows.next()
currentstate = ""
# Iterate through the rows in the cursor
#
while row:

if currentstate != row.Site_Name:
currentstate = row.Site_Name
print "Site: %s," % (row.Site_Name)
exp = "FID" + " = " + str(row.getvalue("Fid"))
print exp
# Process: Make Feature Layer...
gp.MakeFeatureLayer_management(Possible_Destinatio ns, "Dest")
# Process: Select Layer by Attribute...
gp.SelectLayerbyAttribute_management("Dest","NEW_S ELECTION",exp)
print "Selected Desired Row"
# Process: Cost Path...
tempEnvironment0 = gp.extent
gp.extent = "227845 3922854 315204.999984 3963250.127696"
gp.CostPath_sa("Dest", Cost_Distance_n_, Backlink_n_, CostPath_Output, "EACH_ZONE")
gp.extent = tempEnvironment0
gp.overwriteoutput = 1
print "Cost Path"
# Process: Raster to Polyline...
gp.RasterToPolyline_conversion(CostPath_Output, Vector_CostPath, "ZERO", "0", "NO_SIMPLIFY", "VALUE")
print "Polyline"
# Process: Delete...
gp.Delete_management(CostPath_Output, "")
print "Delete"
# Process: Append...
gp.Append_management(Vector_CostPath,Path_Output," NO_TEST")
print "append"
# Process: Delete (2)...
gp.Delete_management(Vector_CostPath, "")
print "delete"
print "Row Completed"
row = rows.next()
gp.Delete_Management("lyr")
gp.Delete_Management("Dest")
print "inside loop complete"
source = sources.next()
print "Done"

good luck,

Luke
BRGS, formerly UofAkron