topic Re: Why does python suck?: Discontinuity between codes and tools in Python Questions

Why does python suck?: Discontinuity between codes and tools

RichardThurau — Thu, 01 Mar 2012 17:04:28 GMT

Hi Community:
Why is it certain tasks fail in python, but plug the same data, same parameters into a tool, and it runs without issue? I'm partially looking for answers to some specific questions in this post, and partially looking for developers to improve this product.

I'm trying to find some answers on this code I wrote this yesterday to do some basic tabulations.
I've commented to the right where in this code, python crashed.
In the essence of trying to find all the answers, I'll keep my code as messy as I wrote it (although I shortened pathnames-which might be an issue sometimes?):

### System Tools import arcpy, os from arcpy.sa import * from arcpy import env, sa arcpy.CheckOutExtension("Spatial")  ### Declarations------------------------------- arcpy.env.overwriteOutput = True ### Definitions------------------------------------- ## areaTab1 = "X:\\DATA\\Temp.gdb\\trees11_ward_tab" areaTab2 = "X:\\DATA\\Temp.gdb\\trees06_ward_tab"  geog1_sub= "X:\\DATA\\Target_Geog.gdb\\WardPly" geog1= "X:\\DATA\\Temp.gdb\\WardPly_utm" cs= "C:\\Program Files (x86)\\ArcGIS\\Desktop10.0\\Coordinate Systems\\Projected Coordinate Systems\\UTM\\NAD 1983\\NAD 1983 UTM Zone 18N.prj"  Lc06 = "X:\\DATA\\C_2006\\LandCover_2006.img" trees06 = "X:\\DATA\\trees_06"  trees_in= "X:\\DATA\\Tree.gdb\\Tree_merged_022912" trees_vec= "X:\\DATA\\Tree.gdb\\Tree_merged_022912_1" trees_ras =  "X:\\DATA\\Tree.gdb\\Tree_raster_022912"   ##Processes trees06 = Reclassify(Lc06, "Value", RemapValue([[0, "NODATA"], [1, 10], [2, "NODATA"], [3,"NODATA"], [4,"NODATA"], [5,"NODATA"],[6,"NODATA"], [7,"NODATA"]])) ##FAILED: Assuming code doesn't like .img- But, ran perfectly using Reclassify tool. trees06.save("X:\\DATA\\Working\\trees_06")   arcpy.Project_management(geog1_sub, geog1, cs) # Ran successfully arcpy.FeatureClassToFeatureClass_conversion(trees_in, os.path.dirname(str(trees_in)), trees_vec) ## Making a backup copy of data: Failed. Ran in python after adding actual pathname. arcpy.AddField_management(trees_vec, "Ras", "SHORT") #Success arcpy.CalculateField_management(trees_vec, "Ras", 1) # Success arcpy.FeatureToRaster_conversion(trees_vec, "Ras", trees_ras, 1) # Success  TabulateArea(geog1, "OBJECTID", trees_in, "Value", areaTab1, 1) #FAILED: "Field OBJECTID does not exist". Um, yes it does, and tool ran fine using exact parameters from Tabulate Area tool. print "Tab 1 done" TabulateArea(geog1, "OBJECTID", trees06, "Value", areaTab2, 1)#FAILED: "Field OBJECTID does not exist". Um, yes it does, and tool ran fine using exact parameters from Tabulate Area tool. ## arcpy.JoinField_management(geog1, "OBJECTID", areaTab1, "OBJECTID", "VALUE_1") #This line failed first time through, then passed after restart. (?) arcpy.JoinField_management(geog1, "OBJECTID", areaTab2, "OBJECTID", "VALUE_10")# Succes ## FieldList = ["UTC11_Acres", "UTC11_Pct", "UTC06_Acres", "UTC06_Pct", "Chng_Acres", "Chng_Pct"] ## for field in FieldList:     arcpy.AddField_management(geog1, field, "DOUBLE")  rows = arcpy.UpdateCursor(geog1) for row in rows:     row.UTC11_Acres = (row.VALUE_1 * 0.0002471054)     rows.updateRow(row)     row.UTC11_Pct = (row.VALUE_1 / row.Shape_Area) * 100     rows.updateRow(row)     row.UTC06_Acres = (row.VALUE_10 * 0.0002471054)     rows.updateRow(row)     row.UTC06_Pct = (row.VALUE_10 / row.Shape_Area) * 100     rows.updateRow(row)     row.Chng_Acres = (row.UTC11_Acres - row.UTC06_Acres)     rows.updateRow(row)     row.Chng_Pct = (row.UTC11_Pct - row.UTC06_Pct)     rows.updateRow(row)  arcpy.DeleteField_management(geog1, ["VALUE_1", "VALUE_10"])

Most of the issues I'm having and have had in the past are dealing with rasters. I've received some good feedback on how to deal with some issues, but are there things I can be doing differently to ease my woes?

I'm trying to convince my boss that python is a lot more efficient than clicking or modeling in ModelBuilder. It's a hard sell when i have to restart a script 8 times to get it to run.

General comments and specific resolutions are much appreciated.

Thanks!

Rich

Re: Why does python suck?: Discontinuity between codes and tools

ChrisMathers — Thu, 01 Mar 2012 17:54:18 GMT

Run the Tabulate Area tool from ArcMap and in the results window right click and choose "Copy as python snippit." This will give you "perfect" syntax.

Re: Why does python suck?: Discontinuity between codes and tools

RichardThurau — Sat, 11 Dec 2021 21:40:24 GMT

All the code here has been used successfully in other cases. Sometimes success comes from the Fail, save, restart, Pass cycle.
I haven't used the snippet option very often.

I wonder why the snippet code looks so different than the way Help describes how to write? For example:

Snippet:

arcpy.Reclassify_sa(Lc06, "Value","0 NODATA;1 10;2 NODATA;3 NODATA;4 NODATA;5 NODATA;6 NODATA;7 NODATA", trees06,"DATA")

Code written from Help:

trees06 = Reclassify(Lc06, "Value", RemapValue([[0, "NODATA"], [1, 10], [2, "NODATA"], [3,"NODATA"], [4,"NODATA"], [5,"NODATA"],[6,"NODATA"], [7,"NODATA"]])) 
trees06.save("X:\\DATA\\Working\\trees_06")

The snippet code fails, btw: "AttributeError: 'module' object has no attribute 'Reclassify_sa'" - I guess that's why the Help articles are there.

I'll try using the suggestion in other cases though.

Thanks

Rich

Re: Why does python suck?: Discontinuity between codes and tools

DanPatterson_Retired — Thu, 01 Mar 2012 19:25:23 GMT

within the help file code there is one salient line

from arcpy.sa import *

was that included in your failed script? Otherwise you may have to checkoutExtension if you aren't using an ArcInfo license

Re: Why does python suck?: Discontinuity between codes and tools

curtvprice — Thu, 01 Mar 2012 19:32:00 GMT

I wonder why the snippet code looks so different than the way Help describes how to write?

The snippet example used the string representation of the argument (this is what is returned BTW by getParameterAsText()), the example in the help uses python structures like lists. Both argument formats are supported.

In the tool and modelbuilder, the GUI usally handles all this text-object conversion for you -- when you're scripting you need to be very careful how you format arguments so the tool will see the correct object representation.

The snippet code fails, btw: "AttributeError: 'module' object has no attribute 'Reclassify_sa'"

I think this may be a bug in the copy as python snippet - as this syntax worked with 9.3 but won't work with 10.0. Most spatial analyst tools must be accessed using" rel="nofollow" target="_blank">http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//00p600000002000000.htm]using Python map algebra in 10.0:

arcpy.CheckOutExtension("spatial")
outgrid = arcpy.sa.Reclassify(...)

or

arcpy.CheckOutExtension("spatial")
from arcpy.sa import *
...
outgrid = Reclassify(..,)

I'm trying to convince my boss that python is a lot more efficient than clicking or modeling in ModelBuilder. It's a hard sell when i have to restart a script 8 times to get it to run.

IMHO modelbuilder is a great macro tool for your work, but when you need to build a tool that needs full validation and documentation and error messages, python is the way to go.

Re: Why does python suck?: Discontinuity between codes and tools

RichardThurau — Thu, 01 Mar 2012 19:36:11 GMT

Yes Dan. I just commented out the reclassify line from the code I posted above and tried the snippet. arcpy.CheckOutExtension("Spatial") is listed too.

Sounds like it is possible to write a reclassify in such a way? Seems to make more sense than the Help method, having to declare RemapRange or RemapValue and then saving the output.

Thanks

RT

Re: Why does python suck?: Discontinuity between codes and tools

MikeMacRae — Thu, 01 Mar 2012 21:12:24 GMT

I was having the same issue a week or so ago. One of the tests I ran was to print out my parameters which included path names. I had something similar to your path for the .img in that I had a folder called '2009' which contains integer values. I was getting weird print lines where the folder was, so I could tell that the path wasn't being read properly. I changed all my pathway parameters to raw string and they printed out fine, so in your case:

Lc06 = r"X:\DATA\C_2006\LandCover_2006.img"

I think when I set it up as a parameter, I set it to arcpy.GetparameterAsText() it was doing the raw string conversion for me, whereas you have to set it in idle. When I ran from idle, it crapped out until I changed to raw. This might be your issue. You might want to change the rest of your parameters to raw string as well.

Cheers,
Mike

Re: Why does python suck?: Discontinuity between codes and tools

RichardThurau — Fri, 02 Mar 2012 15:25:09 GMT

Everyone,
Honestly, I was pretty disgruntled when I started this post, but thanks to some great offerings I feel like I've learned a bit about what's happening. I'll be applying your suggestions a bit later today and report on the result.

For now, I just want to say Thanks for all your input!

Rich

Re: Why does python suck?: Discontinuity between codes and tools

StacyRendall1 — Fri, 02 Mar 2012 19:29:24 GMT

Richard,

It's not Python that you should be angry at - it's Arcpy (i.e. the 'site-package' developed by ESRI that lets you use Python to access Arc functionality). Usually with programming languages if something goes wrong, it's the programmers fault (for making silly mistakes, etc.). However with Arcpy there are continual issues with data locks, odd/variable syntax, inconsistent function name capitalization, etc., etc., etc. These make it particularly hard to learn, and the documentation is not always the greatest...

Copy As Python Snippet bug (Spatial Analyst 10.0)

curtvprice — Sat, 03 Mar 2012 13:58:54 GMT

I think this may be a bug in the copy as python snippet.

I contacted Esri Support and, sure enough:

You are correct the Copy As Python Snippet for the Con tool is not generating the correct syntax. I have submitted a software bug report on your behalf for this issue. The reference number and subject line for this bug is as follows:
[#NIM078805 The Spatial Analyst > Conditional > Con Python snippet from Results window uses incorrect syntax.]

Re: Why does python suck?: Discontinuity between codes and tools

RichardThurau — Sat, 03 Mar 2012 16:20:07 GMT

Stacy,
I absolutely agree and I should have described more specifically, where my emotional dissatisfaction is aimed. Coding is really a relief from the uncertainties that are inherent in windows computing. I work a lot with R and less with Python, but with both applications I know when something runs, it will deliver what it is supposed to, or I have to figure out where I screwed up :-j
My gripes are with arcpy. But, i guess it does a lot, and generally the documentation available blows away anything else available. Certainly part of my problem is just not being a programmer, but a spatial analysis guy who is now trying to use arcpy for spatial analysis.

My goal is to start using open source python spatial analysis packages like GDAL/OGR. I just don't know if these will be more reliable or not. I'm guess so, mostly because of the things you've described in your previous post.

Curtis, thanks for looking into that. I've heard 10.1 is getting some major arcpy reworking, so I'm curious to know how many of these issues might be corrected there.

Thanks everyone for your thoughts and comments.

Rich

Re: Why does python suck?: Discontinuity between codes and tools

StacyRendall1 — Sat, 03 Mar 2012 20:22:55 GMT

Hi Richard,

I have done a bit in GDAL/OGR (just with vectors) a few years ago. The documentation is appalling (it exists in full for C/C++ versions of GDAL/OGR, then sometimes there are bits at the bottom of the documentation page showing something similar in Python), although there is a pretty good community if you get stuck. Just remember to use print dir() all the time, to find out what methods are available for different objects. I think GDAL/OGR can sometimes be a bit faster overall (some of the processes may be similar speed or slower, but it is a lot quicker to load and basic things (opening files, etc.) are a hell of a lot faster. This guy has a blog that I referred to quite a lot when using it, and I'm hoping to share some of my GDAL/OGR experiences on my blog at some stage too. GDAL/OGR is a bit limiting as there is no support yet for File Geodatabases (ESRI only recently opened up the standard), so if you do vector stuff you have to use shapes or Personal Geodatabases or something.

Cheers,
Stacy

Re: Why does python suck?: Discontinuity between codes and tools

RichardThurau — Tue, 06 Mar 2012 18:59:56 GMT

Stacy, thanks a lot for the advice and info. I probably need to get my python in a little bit better form, but someday soon I'd like to be able to reduce my dependence on ESRI.

All in all, using the raw directories seems to have reduced a lot of issues for me.

Thanks again to everyone for your help.

Rich

Re: Why does python suck?: Discontinuity between codes and tools

curtvprice — Tue, 06 Mar 2012 19:11:59 GMT

My goal is to start using open source python spatial analysis packages

There's some really good stuff in R. And of course you can run" rel="nofollow" target="_blank">http://rpy.sourceforge.net/]run R from Python.

CRAN Task View: Analysis of Spatial Data

Re: Why does python suck?: Discontinuity between codes and tools

StacyRendall1 — Wed, 14 Mar 2012 19:32:32 GMT

Big news - just found out that GDAL/OGR 1.9 is out, which includes read/write for File Geodatabases! Don't have time to test it right now, but I hope to give it a blast some time in the next few months...

If you are unsure about downloading/installing it (the Python site has the new version, but pretty indecipherable instructions), check out this post.

Re: Why does python suck?: Discontinuity between codes and tools

MichaelStead — Thu, 15 Mar 2012 02:37:40 GMT

If you are installing GDAL remember that you have 32 bit python installed with ArcGIS. I didn't want to risk confusing Arc so I didn't attempt to install an additional 64 bit version. Also, Arc's python for me was 2.6.5, but the python bindings installer didn't see it. This had to do with being installed for "All Users" and Windows 7 only making a registry in Local_Machine and not Current_User. Just found the entries and exported to a reg file, opened in notepad, find/replace LOCAL_MACHINE with CURRENT_USER, save, write new file to registry. You'll have lots of python entries so you might have to google it if you run into this. Common problem to other python dependent software and windows 7.

Re: Why does python suck?: Discontinuity between codes and tools

StacyRendall1 — Thu, 15 Mar 2012 20:48:50 GMT

Thanks for your tips Michael. I most often use two different PCs, one Windows 7 and one Windows XP, both have two versions of Python installed (Arc's 2.6.5 (x32) and 3.2.whatever (x64)) and both have the appropriate GDAL installed. However, you can only use one GDAL install at a time, as the GDAL PATH variable needs to be changed accordingly. Perhaps because my Windows 7 computer is a private one I have never had the issue you mentioned!

Re: Why does python suck?: Discontinuity between codes and tools

RichardThurau — Mon, 19 Mar 2012 03:54:49 GMT

Great, thanks for lowdown on GDAL. I'm hoping to have time to mess with it in a couple weeks.

Curtis, the R stuff is awesome! I struggle to figure out how to manipulate data in Python like I can in R.

For example, right now I'm trying to figure out how to take a list of tables in a geodatabase and combine the contents into a single table. In R with common column names the command is rbind. In Python maybe I'd need to use vstack...

I'll likely start a new post for help with this one, but I'd be up for feedback either through python or Rpy.

Thanks everyone!

RT

Re: Why does python suck?: Discontinuity between codes and tools

curtvprice — Mon, 19 Mar 2012 15:06:52 GMT

For example, right now I'm trying to figure out how to take a list of tables in a geodatabase and combine the contents into a single table. In R with common column names the command is rbind. In Python maybe I'd need to use vstack...

If the tables are in a geodatabase already, why not just use the Merge tool? The field mapping object gives you an awful lot of extra functionality - for example, if the column names don't match you can build mappings.

Re: Why does python suck?: Discontinuity between codes and tools

RichardThurau — Mon, 19 Mar 2012 20:55:00 GMT

Curtis,
Yes, I actually ended using the Append command.

My poorly worded question is regarding how to move an arcpy object, like an attribute table, into an R object like a matrix, table, or dataframe. This is the missing link for me, since I feel pretty comfortable with R, but not so much with Python. It would be awesome to take a list of attribute tables, move them to R, manipulate the tables, then save them back as attribute tables again. Might be getting too complicated without a specific example which i do not have time to produce right now.

Thanks for your suggestion.

Rich