Latest Contributions by curtvprice

Sorry.  I forgot there are several iterator choices.  Yes, it is a feature selection iterator that would iterate selections based on your owner name values. Desktop 10.1 Help: Iterate Feature Selection ModelBuilder @ 10.x has become a very useful tool for personal workflows like this! ... View more

Hi Peter! Yes, that's a big raster: 34651 * 24934 * 4. / ( 1024 ** 3) > 3.2 GB uncompressed. The fact that the tools are not failing is a great credit to Esri's GP team. (Well done, folks!) I've used the Flow Accumulation tool within Arc Hydro, the Flow Accumulation tool within the Spatial Analyst Extension; Hydrology Toolbox as well as tried running it from Python hoping for better processing time. In 10.x, the same actual raster tools run, whether you initiate it from a tool, model or Python, or even ArcObjects "Op" interfaces. (Although this takes away some options from 9.x, I'm happy to be free of the complications-- although if desperate you can still get to MOMA - with its many limitations -- through arcpy.gp.) So I don't expect any improvement there. The Flow Accumulation algorithm that is being used seems to be hard coded in that it doesn't use any more than 2 GB memory when processing the Flow Direction GRID to generate the Flow Accumulation GRID. This has to do with that the tools are developed and usually run in 32-bit, so you are limited with how much direct-access memory can be addressed at one time. Two things I suggest trying (using a small test area!!): 1) If you have 10.1 SP1 installed, and are running Win7-64, try running the tool using x64 Background Geoprocessing. (Or more directly by running your script from 64-bit Python/arcpy.) It's possible the Flow Accumulation tool has been modified in 64 bit to use more RAM. 2) Test using other raster data types and see if the performance varies. Since GIS processing is often I/O dependent, different datatypes sometimes work better with different tools, due to how the process interacts with the particular data structure. Candidates I'd compare first would be grid, file gdb, .img, and .tif. (For example, I have heard that Flow Direction is quite zippy for .tif output.) For best performance (you just be doing a lot of thrashing between memory and disk) you may be forced to tile your processing into basins (by building watersheds from your flow direction raster). The NHDPlus folks have taken this approach when hydro-processing DEMs for the continental U.S. Hope this is helpful. I'd look further into the NHDPlus docs to see if anything in their approach can help you. ... View more

Hi Peter! Yes, that's a big raster: 34651 * 24934 * 4. / ( 1024 ** 3) > 3.2 GB uncompressed. The fact that the tools are not failing is a great credit to Esri's GP team. (Well done, folks!) I've used the Flow Accumulation tool within Arc Hydro, the Flow Accumulation tool within the Spatial Analyst Extension; Hydrology Toolbox as well as tried running it from Python hoping for better processing time. In 10.x, the same actual raster tools run, whether you initiate it from a tool, model or Python, or even ArcObjects "Op" interfaces. (Although this takes away some options from 9.x, I'm happy to be free of the complications-- although if desperate you can still get to MOMA - with its many limitations -- through arcpy.gp.) So I don't expect any improvement there. The Flow Accumulation algorithm that is being used seems to be hard coded in that it doesn't use any more than 2 GB memory when processing the Flow Direction GRID to generate the Flow Accumulation GRID. This has to do with that the tools are developed and usually run in 32-bit, so you are limited with how much direct-access memory can be addressed at one time. Two things I suggest trying (using a small test area!!): 1) If you have 10.1 SP1 installed, and are running Win7-64, try running the tool using x64 Background Geoprocessing. (Or more directly by running your script from 64-bit Python/arcpy.) It's possible the Flow Accumulation tool has been modified in 64 bit to use more RAM. 2) Test using other raster data types and see if the performance varies. Since GIS processing is often I/O dependent, different datatypes sometimes work better with different tools, due to how the process interacts with the particular data structure. Candidates I'd compare first would be grid, file gdb, .img, and .tif. (For example, I have heard that Flow Direction is quite zippy for .tif output.) For best performance (you just be doing a lot of thrashing between memory and disk) you may be forced to tile your processing into basins (by building watersheds from your flow direction raster). The NHDPlus folks have taken this approach when hydro-processing DEMs for the continental U.S. Hope this is helpful. I'd look further into the NHDPlus docs to see if anything in their approach can help you. ... View more

Clearly you do have 32-bit paths in your sys.path that do not belong there when you are running the 64-bit python. The question is where they are coming from. This sure looks like something you'd want to open a support incident to get to the bottom of the issue. ... View more

I would expect your expression to look more like: "rastercalc" + "rastercalc2" + "rastercalc3" with the three names showing up in the Raster Calculator tool dialog box. I'm assuming the three names are raster datasets or raster layers that are elements in the model. I'll also mention that the above expression would probably be handled easier (and more efficiently, and with handy additional option on how NoData cells are handled) using the Cell Statistics tool instead of Raster Calculator. ... View more

Here's my updated ListUnique script - works with 9x/10x. Thanks guys, this is a big improvement to what I had before. A cute tweak I learned that sorted(set()) returns a list. def ListUnique(inTable,Field):     """Create a list of unique values from a table/tableview.     arguments     inTable  Table or table view     Field    Field name     """     Row, Rows = None, None     try:         try:             # this will only work for 10.1             import arcpy.da             lstValues = \                 sorted({r[0] for r in                     arcpy.da.SearchCursor(inTable, [Field])})         except:             import arcgisscripting             gp = arcgisscripting.create(9.3)             Rows = gp.SearchCursor(inTable,"","",Field,Field)             Row = Rows.next()             lstValues = []             while Row:                 lstValues.append(Row.getValue(Field))                 Row = Rows.next()             # unique-ize and sort the list             lstValues = sorted(set(lstValues))         return lstValues     except:         raise     finally:         if Row: del Row         if Rows: del Rows ... View more

Hey Curt, I thought about doing a MakeFeatureLayer and then running a series of SelectLayerbyAttributes but from my experience, that significantly increases processing time. Not a big deal with small to moderate sized datasets, but with 3.2 million records, I didn't want to play with those implications. I agree - I fixed the code above so you're just selecting off the layer directly with the Select_analysis tool. If you added a field index you'd get the time back in spades when you ran the Select tool -- even faster! Someone commented on the post about leveraging the TableToNumPyArray function to get better performance Talk about gilding the lily. arcpy.da.SearchCursor is pretty darn fast. It's so fast, I wonder if Esri's team they used a numpy array to do it! ... View more

I was able to pull up a list of the unique store IDs by using: def Unique_Values(table, field):     with arcpy.da.SearchCursor(table, [field]) as cursor:         return sorted({row[0] for row in cursor}) fc = "C:\Users\username\Documents\ArcGIS\Default.gdb\MyCustomers" field = "DID" UniqueValues (fc, field) which resulted in a list of all of my unique store IDs, 957 in all. Neat. I may steal that. What I'm not sure of however, is how to go about iterating through that list and using each value in the list in a where_clause in a "FeatureClassToFeatureClass_conversion" to export each unique dataset. Note, you could avoid scripting altogether and do this in ModelBuilder using Iterate Feature Selection. But, if you want to do this in Python, it's a matter of making a layer and selecting on it: lyr = arcpy.CreateFeatureLayer(management(fc, "lyr") stores = UniqueValues(lyr, field) for store in stores:     where = "{0} = {1}".format(field, store)     outFC = "store{0}".format(store)     arcpy.Select_analysis(lyr, outFC, where) UPDATE: Chris beat me to the post. Nice one-liner, Chris!  Just a note about my more verbose approach: If a GP tool gets a layer input, it can assume that the input exists and its field list is already in memory ready to roll. This can make a difference if you need to do 957 select/copy operations. ... View more

And it looks like I can use the 'Single Output Map Algebra' tool to place this in ModelBuilder. Yes, indeed. Map Algebra is quite useful! For the thread I should add that when you move up to 10.x SOMA and Raster Calculator (from the menu) are no longer available. Your 9.x model that uses SOMA will still run, you just will have to do it differently in new models. What's new in ArcGIS Spatial Analyst 10 ... View more

Scott: Warning: your code has mixed indents, some tabs, some spaces. It's best to use one or the other, I like spaces.  This can lead to unexpected results as your indents may not work exactly as intended. Most Python IDEs have a setting to convert automatically to spaces for you. ... View more

If you used the Define Projection tool, or modified the coordinate system properties in a dialog box, that does not change the coordinates, it just changes the metadata that tells ArcGIS what projection the coordinates reference. So for those shapefiles that you altered, the coordinates are still in the original projection, but ArcGIS thinks they are in the new UTM zone so draws them in the wrong place. You need to project the datasets from 18N to 17N using the Project tool. Hope this helps. Update: Hi Mark - you responded a few seconds before I did! ... View more

Here's a better description of using pth files from the gis.stackexchange.com site, posted by  my colleague Rich Signell. If you don't want to tweak your environment for your python session, this is an easier approach than a .bat file. Note path files are read in alpha order, so if you name your custom pth file starting with "zz" it will be loaded last -- which may avoid conflicts. [/hr] The Enthought Python Distributions, including Canopy, are stand-alone and don't need anything in the registry or in the environment variables. So they can play nice with other Python distributions like the one that comes with ArcGIS 10.1. To access ArcGIS 10.1 python modules from Enthought Canopy: Create a "path" file in the Canopy user directory that points to the ArcGIS 10.1 python modules. On my system, I created a text file called arcpy.pth with these 4 lines: # .pth file for arcpy C:\ArcGIS\Desktop10.1\bin C:\ArcGIS\Desktop10.1\arcpy C:\ArcGIS\Desktop10.1\ArcToolbox\Scripts and put arcpy.pth in the Canopy User site-packages directory: C:\Users\rsignell\AppData\Local\Enthought\Canopy32\User\lib\site-packages To access Enthought Canopy modules from ArcGIS 10.1: Create a "path" file in the Arc site-packages folder that points to the Canopy python modules. On my system, I created a text file called epd.pth with these 3 lines: # .pth file for EPD Canopy C:\Users\rsignell\AppData\Local\Enthought\Canopy32\User\Lib\site-packages C:\Users\rsignell\AppData\Local\Enthought\Canopy32\System\Lib\site-packages and then put epd.pth in the Arc site-package folder: C:\Python27\ArcGIS10.1\Lib\site-packages This works because Enthought Canopy and ArcGIS10.1 are at the same version of Python (2.7) and Numpy (1.6). I hope we can continue to have this compatibility! ... View more