POST
|
Is anyone aware of issues with ArcGIS 10 SP 5 with ArcHydro version 2.0.1.133 (running on a Win 7 x64 workstation ) and any of its tools? It seems I have discovered a problem with the Batch Watershed Delineate for Polygons and I have no clue if other tools in the ArcHydro toolbar suffer from similar or other issues in the aforementioned environment / configuration. I was attempting to delineate the contributing area for a lake in Kansas. I prepped the data as I had done previously for 50+ other sites using ArcHydro 1.4 with ArcGIS 9.3, so workflow isn't the issue. The DEM is 30m and projected (equal area)...but when I tried to delineate using the Batch Watershed Delineate for Polygons tool, it would not produce a hydrologically correct watershed. So, after trying everything I could think of and also having no luck with a 10m DEM on my workstation, my last shot was to try it on 9.3. We have a considerably older / less powerful desktop used for running our plotter as it has ArcGIS 9.3 (SP1 with ArcHydro 1.4) on it with Win XP x32. I tried to delineate the watershed with the exact same data I prepped on my workstation mentioned above, and it worked. See the attached image as a PNG file - the orange is the lake, and the purple is what ArcHydro spit out as the junk watershed on my workstation running 10, clearly has problems. The larger green watershed is the hydrologically correct watershed generated on the 9.3 machine. Again, same input data used for both attempts. I looked around online for a few minutes and could not find any known bug lists for ArcHydro. I also tried the newest version of ArcHydro version 2.1.0.116 posted on the ESRI FTP, and it didn't even generate a junk watershed...but this time it coded a "-1" which indicates an error in the BatchDone field that it adds to the feature class containing the polygon(s) to be delineated. So, even less output and worse off. [ATTACH=CONFIG]17683[/ATTACH] Any info would be appreciated thanks Tom
... View more
09-13-2012
02:46 PM
|
0
|
0
|
2643
|
POST
|
It appears the gp.LoadSettings method does not work - I saved out the ArcCatalog ArcToolBox Environment settings as an xml file, and when I try to load the settings in the script, it says the object cannot be found even though I triple verified I had the path and file name correct. I found this forum post that discusses this in the context of ArcObjects but seems the Python approach does the same thing: http://forums.arcgis.com/threads/7204-igeoprocessor.savesettings-and-igeoprocessor.loadsettings-not-persisting-settings However, I did set the gp.ScratchWorkspace this time and that seems to have made the difference. I never would have thought that would impact the performance that much, I simply specified the same scratch workspace I have set in the ArcToolBox Environment. That's the only change I made in my testing configuration and that seems to have done it. The script ran from the command (using all the same input and output file settings and files) in about 6 minutes 49 seconds, actually about 40 seconds quicker than the ArcCatalog run I did. I verified the watersheds and all file output and looks to be spot on. thank you, sir! Tom
... View more
04-12-2011
08:12 AM
|
0
|
0
|
346
|
POST
|
Hi- I know it's not instantiating the gp that's an issue - I have timestamps for when a particular tool starts, and when it finishes. Watershed and Snap Pour Point are the ones with terrible performance (I've build in a lot of print statements and writing to a log .txt file the timestamps for post-run analysis of where the time suck(s) are). I'm using the ArcToolBox Spatial Analyst Hydrology tools, not Toolbar tools. In both runs (python script and manually in arccatalog) I'm outtputing to and inputting from the same personal geodatabases - they are not crammed with data so I'm not pushing a data size limit or anything. The elevation derivative rasters are in the IMAGINE image format in the pgdb's. How can you change what you mention regarding current and scratch workspaces? I noticed in my Windows user profile Temp directory (C:\Documents and Settings\...\Local Settings\Temp) that there are some of the g_g_g2 files (or similar) in there. Just to see what happens, I set up a new ArcToolbox Toolbox and added my python script as a script that I could run. I have the script running based on argument input from a "configuration" text file that it reads in, so in this fashion I just double click on the script and click "OK" as there are no variables to set. The script ran in 7 minutes 32 seconds vs. 20+ minutes for my same three test sites, one-third of the total time. It definitely shaved off a bunch of time on the Snap Pour Point and Watershed tool runs (everything else was already reasonably fast so not worried about other tools/functions). I ran the script via ArcToolBox in ArcCatalog, not ArcMap. SO....it seems that there is something going on (or perhaps in this case not going on) when you run a python script outside of ArcGIS (via command line or an IDE) that incurs considerable overhead on raster-calculation tools. I don't have any special raster settings in the ArcToolBox environments. As far as I know, these are the defaults (I've never messed with them - see attached screenshot). How would these (or a lack of these defaults) impact such tools - is that the issue? If so, can they be set in standalone python code?? thanks- I think this is getting closer Tom
... View more
04-11-2011
01:53 PM
|
0
|
0
|
346
|
POST
|
Hi, My question is: what might cause the ArcGIS 9.3 Hydrology Toolbox Snap Pour Point and Watershed tools to run *significantly* slower in a Python script versus manually running the tools in ArcCatalog (both runs using the exact same input data files and specification parameters)?? I'm running this script from the windows command line so neither ArcMap or ArcCatalog are open; using ArcGIS 9.3.1 on a Windows XP machine with 4 GB of RAM and plenty of processor power (a Dell Precision machine); and all input data are local. I have pre-processed Flow Accumulation, Flow Direction, and a Feature Class of water-quality sample sites to be used as pour points, and all datasets are in NAD83 UTM 17N projection. The Flow Accum and Flow Dir data files have spatial extents based on Hydrologic Unit Code (HUC) eight digit boundaries. The script iterates through each record in the sample site feature class, determines which eight digit HUC the point falls in, selects and exports the site record as a new feature class in a scratch geodatabase, and then handles the watershed processing using that exported point feature class. I've stripped out comments and some other calls to a "processing log timestamp" function for reporting tool run times just to focus here on the question at hand. The biggest time sinks based on the processing log reporting are the Snap Pour Point tool and Watershed tool. We're talking on the order of 4+ minutes per site using a subset of all the sites for testing, and then 3-4 minutes for each site with the Watershed tool. However, I can take the same single point feature class and the same flow accumuluation and flow direction rasters that the Python script is using, and run the Snap Pour Point and Watershed tools manually in ArcCatalog in approx 1 minute or less for each site (the resulting watersheds are the same as the watersheds the script produces). Example: one of the sites I manually ran the Snap Pour Point tool in 58 seconds via ArcCatalog, while the Python script executing that tool took 3 minutes 59 seconds for the tool to complete. I then ran the Watershed tool manually for the same site and it took 1 minute 1 second in ArcCatalog, while the Python script executing the Watershed tool took 4 minutes 8 seconds for the tool to complete. I'm using a 50 meter buffer for the Snap Pour Point tool in both the Python script and during my ArcCatalog manual run comparisons. As you can see, CONSIDERABLY slower in the Python script. I've hit a wall with what I can figure out, so much appreciate any suggestions or insight. thanks Tom try:
if hucGdb in gdbFiles:
outScratch = scratch + '\\' + delinSiteID + 'temp'
flowAccum = filePath + '\\' + hucGdb + '\\' + 'FA_' + huc
flowDir = filePath + '\\' + hucGdb + '\\' + 'FD_' + huc
tempEnvironment = gp.extent
gp.extent = flowAccum
clause = '[DELIN_ID] = ' + "'" + delinSiteID + "'"
gp.Select_analysis(delinSites, outScratch, clause)
gp.extent = tempEnvironment
outSnapRaster = snapPoint + '\\' + 'p' + delinSiteID
gp.SnapPourPoint_sa(outScratch, flowAccum, outSnapRaster, tolerance)
gp.extent = tempEnvironment
outSnapFeature = snapPoint + '\\' + 'p' + delinSiteID + '_snap'
gp.RasterToPoint_conversion(outSnapRaster, outSnapFeature, "VALUE")
gp.AddField_management(outSnapFeature, "ORIGINAL_SITE_ID", "TEXT", "#", "#", "20")
gp.CalculateField_management(outSnapFeature, "ORIGINAL_SITE_ID", '"' + realID + '"')
gp.AddField_management(outSnapFeature, "DELIN_ID", "TEXT", "#", "#", "20")
gp.CalculateField_management(outSnapFeature, "DELIN_ID", '"' + delinSiteID + '"')
outWsRaster = scratch + '\\' + delinSiteID
tempEnvironment = gp.extent
gp.extent = flowDir
gp.Watershed_sa(flowDir, outSnapRaster, outWsRaster)
gp.extent = tempEnvironment
outWsFeature = basinOutput + '\\' + delinSiteID
gp.RasterToPolygon_conversion(outWsRaster, outWsFeature, "SIMPLIFY")
gp.AddField_management(outWsFeature, "DELIN_ID", "TEXT", "#", "#", "25")
gp.CalculateField_management(outWsFeature, "DELIN_ID", '"' + delinSiteID + '"')
gp.AddField_management(outWsFeature, "ORIGINAL_SITE_ID", "TEXT", "#", "#", "25")
gp.CalculateField_management(outWsFeature, "ORIGINAL_SITE_ID", '"' + realID + '"')
else:
msg = """Site ID """ + realID + """ does not have a geodatabase of
data, moving onto the next site"""
print msg
logFile.write(msg + '\n')
except:
print "ARCGISSCRIPTING ERROR: " + gp.GetMessages(2)
msg = "ARCGISSCRIPTING ERROR: " + gp.GetMessages(2)
logFile.write(msg + '\n')
print "ERROR: " + ErrorDesc.message
msg = "ERROR: " + ErrorDesc.message
logFile.write(msg + '\n')
row = rows.next()
... View more
04-11-2011
09:59 AM
|
0
|
4
|
791
|
POST
|
Hi, I'm trying to determine how to delineate a wateshed for a polygon (example: lake polygon) so I can automate using Python scripting. ArcHydro has a "Batch Watershed Delineation from Polygons" tool that works well, but I don't believe the Watershed Processing tools of ArcHydro have been ported into geoprocessing like the ArcHydro Terrain Preprocessing tools have. I don't want to use pour points for the lakes to delineate because the data I'm dealing with are sediment cores taken in the actual lake itself, so I need to use a polygon of the lake. How could one do this using the Spatial Analyst hydrology tools and/or the ArcHydro Terrain Preproccessing tools available in the Spatial Analyst toolbox? I'm sure it's possible, I just haven't figured out how to do this... thanks Tom Burley USGS, Austin TX
... View more
09-29-2010
09:48 AM
|
0
|
2
|
1608
|
POST
|
Ken, that worked, just tested it on my data. I would have never figured that out. Yeah I'm not familiar with C so that was completely new for me - this is really the first week I've messed with Python substantially. I'm working through the online free Dive Into Python book and have been dinking around with some stuff I've wanted to automate with geoprocessing like this. Do you have any online references for this? I don't quite understand what's going on with the Expression that's now being references via the codeblock parameter I just did a quick google search and found this: http://www.python.org/dev/peps/pep-0238/ Does that cover the gist of what you changed and what I was missing? I didn't see any references to the function you include after the from __future__ part with the realDiv function - what does that do? That's just handling the passed-in Count field variable as the numerator and the x sum variable as the denominator? Is that a necessary item to include or would the future import division statement as described in the above url be sufficient? thanks much for your help - this was driving me insane Tom
... View more
09-02-2010
01:44 PM
|
0
|
0
|
428
|
POST
|
Hi Ken, I tried that but am still not getting success. I checked to verify that the COUNT field is numeric. The script seems to run OK, but I go and check the PROPORTION field and all the values are zero. I tried just assigning the value of the X variable (no calculation) and that worked, there's just something not working with determining the quotient of the COUNT field records based on the total stored in the variable X. Here's my code again, and attached are the rasters I'm trying in a personal GDB:
import sys
import string
import os
try:
#for 9.2 and above
import arcgisscripting
gp = arcgisscripting.create(9.3)
gp.AddMessage("\n" + "Using ArcMap 9.2 or above with arcgisscripting..." + "\n")
except:
#for 9.0/9.1
import win32com.client
gp = win32com.client.Dispatch("esriGeoprocessing.GpDispatch.1")
gp.AddMessage("\n" + "Using ArcMap 9.0/9.1 or above with win32com.client.Dispatch..." + "\n")
#
# Check for the necessary product
Licensed = gp.ProductInfo()
gp.addmessage("You are using the following product " + str(Licensed))
try:
gp.SetProduct(str(Licensed))
gp.addmessage("Successfully set the product type to " + str(Licensed))
except:
gp.addmessage("Could not find an available ArcGIS License")
# Load required toolboxes...
gp.toolbox = "management"
# Set the Workspace
gp.Workspace = sys.argv[1]
Workspace = str(gp.Workspace)
Workspace = string.replace(Workspace, "\\", "/")
# Set the Output Folder
OutFolder = sys.argv[2]
OutFolder = str(OutFolder)
OutFolder = string.replace(OutFolder, "\\", "/")
LogFile = open(OutFolder + "/" + "log.txt", "w")
try:
#Generate a list of the grids in the grid folder
rasters = gp.ListRasters("*", "ALL")
#Iterate through each raster, add a new field called "Proportion", calculate the proportion of land cover type based on those two fields
for raster in rasters:
try:
#Add the new Proportion field
gp.addfield (raster, "PROPORTION", "LONG", "2")
#Use the searchcursor method to sum the values of the count field for calculating the proportions
x = 0
rows = gp.SearchCursor(raster)
row = rows.Next()
field = "COUNT"
while row:
x += row.getvalue(field)
row = rows.next()
print str(x) + " " + raster
gp.calculatefield(raster,"PROPORTION","!COUNT! / " + long(x),"PYTHON")
#rows = gp.SearchCursor(raster)
#row = rows.Next()
LogFile.write("Just finished processing" + raster + "\n")
except:
gp.AddMessage(gp.GetMessages(2))
ErrorMsg = gp.GetMessages(2)
print ErrorMsg
LogFile.write(ErrorMsg + "\n")
except:
gp.AddMessage(gp.GetMessages(2))
ErrorMsg = gp.GetMessages(2)
logfile.write(ErrorMsg + "\n")
LogFile.close()
thanks for the assistance with this....it's probably something small...hopefully Tom Burley U.S. Geological Survey Austin, TX
... View more
09-02-2010
09:11 AM
|
0
|
0
|
428
|
POST
|
Hi, I have a script that just needs slight tweaking (I hope/think). It iterates through a workspace containing rasters, adds a new field called Proportion, then sums the raster value COUNT field in order to then determine the proportion of each raster value type. This is for land cover data - trying to calculate the proportion of land cover type per raster. I think my syntax in calculate field is not quite right:
import sys
import string
import os
try:
#for 9.2 and above
import arcgisscripting
gp = arcgisscripting.create(9.3)
gp.AddMessage("\n" + "Using ArcMap 9.2 or above with arcgisscripting..." + "\n")
except:
#for 9.0/9.1
import win32com.client
gp = win32com.client.Dispatch("esriGeoprocessing.GpDispatch.1")
gp.AddMessage("\n" + "Using ArcMap 9.0/9.1 or above with win32com.client.Dispatch..." + "\n")
#
# Check for the necessary product
Licensed = gp.ProductInfo()
gp.addmessage("You are using the following product " + str(Licensed))
try:
gp.SetProduct(str(Licensed))
gp.addmessage("Successfully set the product type to " + str(Licensed))
except:
gp.addmessage("Could not find an available ArcGIS License")
# Load required toolboxes...
gp.toolbox = "management"
# Set the Workspace
gp.Workspace = sys.argv[1]
Workspace = str(gp.Workspace)
Workspace = string.replace(Workspace, "\\", "/")
# Set the Output Folder
OutFolder = sys.argv[2]
OutFolder = str(OutFolder)
OutFolder = string.replace(OutFolder, "\\", "/")
LogFile = open(OutFolder + "/" + "log.txt", "w")
try:
#Generate a list of the grids in the grid folder
rasters = gp.ListRasters("*", "ALL")
#Iterate through each raster, add a new field called "Proportion", calculate the proportion of land cover type based on those two fields
for raster in rasters:
try:
#Add the new Proportion field
gp.addfield (raster, "PROPORTION", "FLOAT", "2")
#Use the searchcursor method to sum the values of the count field for calculating the proportions
x = 0
rows = gp.SearchCursor(raster)
row = rows.Next()
field = "COUNT"
while row:
x += row.getvalue(field)
row = rows.next()
print str(x) + " " + raster
gp.calculatefield(raster,"PROPORTION","!COUNT! / float(x)","PYTHON")
#rows = gp.SearchCursor(raster)
#row = rows.Next()
logfile.write("Just finished processing" + raster + "\n")
except:
gp.AddMessage(gp.GetMessages(2))
ErrorMsg = gp.GetMessages(2)
print ErrorMsg
LogFile.write(ErrorMsg + "\n")
except:
gp.AddMessage(gp.GetMessages(2))
ErrorMsg = gp.GetMessages(2)
logfile.write(ErrorMsg + "\n")
LogFile.close()
Specifically this part of the code: gp.calculatefield(raster,"PROPORTION","!COUNT! / float(x)","PYTHON") I'm storing the sum of the COUNT field in the variable "x", and then want to use that in gp.calculatefield to divide the values in the COUNT field by the sum of the values stored in "x" in memory to determine the proportion of each land cover type any help would be greatly appreciated - my programming experience is in VBA and VBscript, but I'm trying to break into Python thanks, Tom Burley U.S. Geological Survey Austin, TX
... View more
09-01-2010
02:35 PM
|
0
|
5
|
2179
|
Online Status |
Offline
|
Date Last Visited |
11-11-2020
02:23 AM
|