Why does python suck?: Discontinuity between codes and tools

RichardThurau · ‎03-01-2012

Hi Community:
Why is it certain tasks fail in python, but plug the same data, same parameters into a tool, and it runs without issue? I'm partially looking for answers to some specific questions in this post, and partially looking for developers to improve this product.

I'm trying to find some answers on this code I wrote this yesterday to do some basic tabulations.
I've commented to the right where in this code, python crashed.
In the essence of trying to find all the answers, I'll keep my code as messy as I wrote it (although I shortened pathnames-which might be an issue sometimes?):

### System Tools import arcpy, os from arcpy.sa import * from arcpy import env, sa arcpy.CheckOutExtension("Spatial")  ### Declarations------------------------------- arcpy.env.overwriteOutput = True ### Definitions------------------------------------- ## areaTab1 = "X:\\DATA\\Temp.gdb\\trees11_ward_tab" areaTab2 = "X:\\DATA\\Temp.gdb\\trees06_ward_tab"  geog1_sub= "X:\\DATA\\Target_Geog.gdb\\WardPly" geog1= "X:\\DATA\\Temp.gdb\\WardPly_utm" cs= "C:\\Program Files (x86)\\ArcGIS\\Desktop10.0\\Coordinate Systems\\Projected Coordinate Systems\\UTM\\NAD 1983\\NAD 1983 UTM Zone 18N.prj"  Lc06 = "X:\\DATA\\C_2006\\LandCover_2006.img" trees06 = "X:\\DATA\\trees_06"  trees_in= "X:\\DATA\\Tree.gdb\\Tree_merged_022912" trees_vec= "X:\\DATA\\Tree.gdb\\Tree_merged_022912_1" trees_ras =  "X:\\DATA\\Tree.gdb\\Tree_raster_022912"   ##Processes trees06 = Reclassify(Lc06, "Value", RemapValue([[0, "NODATA"], [1, 10], [2, "NODATA"], [3,"NODATA"], [4,"NODATA"], [5,"NODATA"],[6,"NODATA"], [7,"NODATA"]])) ##FAILED: Assuming code doesn't like .img- But, ran perfectly using Reclassify tool. trees06.save("X:\\DATA\\Working\\trees_06")   arcpy.Project_management(geog1_sub, geog1, cs) # Ran successfully arcpy.FeatureClassToFeatureClass_conversion(trees_in, os.path.dirname(str(trees_in)), trees_vec) ## Making a backup copy of data: Failed. Ran in python after adding actual pathname. arcpy.AddField_management(trees_vec, "Ras", "SHORT") #Success arcpy.CalculateField_management(trees_vec, "Ras", 1) # Success arcpy.FeatureToRaster_conversion(trees_vec, "Ras", trees_ras, 1) # Success  TabulateArea(geog1, "OBJECTID", trees_in, "Value", areaTab1, 1) #FAILED: "Field OBJECTID does not exist". Um, yes it does, and tool ran fine using exact parameters from Tabulate Area tool. print "Tab 1 done" TabulateArea(geog1, "OBJECTID", trees06, "Value", areaTab2, 1)#FAILED: "Field OBJECTID does not exist". Um, yes it does, and tool ran fine using exact parameters from Tabulate Area tool. ## arcpy.JoinField_management(geog1, "OBJECTID", areaTab1, "OBJECTID", "VALUE_1") #This line failed first time through, then passed after restart. (?) arcpy.JoinField_management(geog1, "OBJECTID", areaTab2, "OBJECTID", "VALUE_10")# Succes ## FieldList = ["UTC11_Acres", "UTC11_Pct", "UTC06_Acres", "UTC06_Pct", "Chng_Acres", "Chng_Pct"] ## for field in FieldList:     arcpy.AddField_management(geog1, field, "DOUBLE")  rows = arcpy.UpdateCursor(geog1) for row in rows:     row.UTC11_Acres = (row.VALUE_1 * 0.0002471054)     rows.updateRow(row)     row.UTC11_Pct = (row.VALUE_1 / row.Shape_Area) * 100     rows.updateRow(row)     row.UTC06_Acres = (row.VALUE_10 * 0.0002471054)     rows.updateRow(row)     row.UTC06_Pct = (row.VALUE_10 / row.Shape_Area) * 100     rows.updateRow(row)     row.Chng_Acres = (row.UTC11_Acres - row.UTC06_Acres)     rows.updateRow(row)     row.Chng_Pct = (row.UTC11_Pct - row.UTC06_Pct)     rows.updateRow(row)  arcpy.DeleteField_management(geog1, ["VALUE_1", "VALUE_10"])

Most of the issues I'm having and have had in the past are dealing with rasters. I've received some good feedback on how to deal with some issues, but are there things I can be doing differently to ease my woes?

I'm trying to convince my boss that python is a lot more efficient than clicking or modeling in ModelBuilder. It's a hard sell when i have to restart a script 8 times to get it to run.

General comments and specific resolutions are much appreciated.

Thanks!

Rich

RichardThurau · ‎03-03-2012

Stacy,
I absolutely agree and I should have described more specifically, where my emotional dissatisfaction is aimed. Coding is really a relief from the uncertainties that are inherent in windows computing. I work a lot with R and less with Python, but with both applications I know when something runs, it will deliver what it is supposed to, or I have to figure out where I screwed up :-j
My gripes are with arcpy. But, i guess it does a lot, and generally the documentation available blows away anything else available. Certainly part of my problem is just not being a programmer, but a spatial analysis guy who is now trying to use arcpy for spatial analysis.

My goal is to start using open source python spatial analysis packages like GDAL/OGR. I just don't know if these will be more reliable or not. I'm guess so, mostly because of the things you've described in your previous post.

Curtis, thanks for looking into that. I've heard 10.1 is getting some major arcpy reworking, so I'm curious to know how many of these issues might be corrected there.

Thanks everyone for your thoughts and comments.

Rich

StacyRendall1 · ‎03-03-2012

Hi Richard,

I have done a bit in GDAL/OGR (just with vectors) a few years ago. The documentation is appalling (it exists in full for C/C++ versions of GDAL/OGR, then sometimes there are bits at the bottom of the documentation page showing something similar in Python), although there is a pretty good community if you get stuck. Just remember to use print dir() all the time, to find out what methods are available for different objects. I think GDAL/OGR can sometimes be a bit faster overall (some of the processes may be similar speed or slower, but it is a lot quicker to load and basic things (opening files, etc.) are a hell of a lot faster. This guy has a blog that I referred to quite a lot when using it, and I'm hoping to share some of my GDAL/OGR experiences on my blog at some stage too. GDAL/OGR is a bit limiting as there is no support yet for File Geodatabases (ESRI only recently opened up the standard), so if you do vector stuff you have to use shapes or Personal Geodatabases or something.

Cheers,
Stacy

RichardThurau · ‎03-06-2012

Stacy, thanks a lot for the advice and info. I probably need to get my python in a little bit better form, but someday soon I'd like to be able to reduce my dependence on ESRI.

All in all, using the raw directories seems to have reduced a lot of issues for me.

Thanks again to everyone for your help.

Rich

curtvprice · ‎03-06-2012

My goal is to start using open source python spatial analysis packages

There's some really good stuff in R. And of course you can run" rel="nofollow" target="_blank">http://rpy.sourceforge.net/]run R from Python.

CRAN Task View: Analysis of Spatial Data

StacyRendall1 · ‎03-14-2012

Big news - just found out that GDAL/OGR 1.9 is out, which includes read/write for File Geodatabases! Don't have time to test it right now, but I hope to give it a blast some time in the next few months...

If you are unsure about downloading/installing it (the Python site has the new version, but pretty indecipherable instructions), check out this post.

MichaelStead · ‎03-14-2012

If you are installing GDAL remember that you have 32 bit python installed with ArcGIS. I didn't want to risk confusing Arc so I didn't attempt to install an additional 64 bit version. Also, Arc's python for me was 2.6.5, but the python bindings installer didn't see it. This had to do with being installed for "All Users" and Windows 7 only making a registry in Local_Machine and not Current_User. Just found the entries and exported to a reg file, opened in notepad, find/replace LOCAL_MACHINE with CURRENT_USER, save, write new file to registry. You'll have lots of python entries so you might have to google it if you run into this. Common problem to other python dependent software and windows 7.

StacyRendall1 · ‎03-15-2012

Thanks for your tips Michael. I most often use two different PCs, one Windows 7 and one Windows XP, both have two versions of Python installed (Arc's 2.6.5 (x32) and 3.2.whatever (x64)) and both have the appropriate GDAL installed. However, you can only use one GDAL install at a time, as the GDAL PATH variable needs to be changed accordingly. Perhaps because my Windows 7 computer is a private one I have never had the issue you mentioned!

RichardThurau · ‎03-18-2012

Great, thanks for lowdown on GDAL. I'm hoping to have time to mess with it in a couple weeks.

Curtis, the R stuff is awesome! I struggle to figure out how to manipulate data in Python like I can in R.

For example, right now I'm trying to figure out how to take a list of tables in a geodatabase and combine the contents into a single table. In R with common column names the command is rbind. In Python maybe I'd need to use vstack...

I'll likely start a new post for help with this one, but I'd be up for feedback either through python or Rpy.

Thanks everyone!

RT

curtvprice · ‎03-19-2012

For example, right now I'm trying to figure out how to take a list of tables in a geodatabase and combine the contents into a single table. In R with common column names the command is rbind. In Python maybe I'd need to use vstack...

If the tables are in a geodatabase already, why not just use the Merge tool? The field mapping object gives you an awful lot of extra functionality - for example, if the column names don't match you can build mappings.

RichardThurau · ‎03-19-2012

Curtis,
Yes, I actually ended using the Append command.

My poorly worded question is regarding how to move an arcpy object, like an attribute table, into an R object like a matrix, table, or dataframe. This is the missing link for me, since I feel pretty comfortable with R, but not so much with Python. It would be awesome to take a list of attribute tables, move them to R, manipulate the tables, then save them back as attribute tables again. Might be getting too complicated without a specific example which i do not have time to produce right now.

Thanks for your suggestion.

Rich