modelbuilder vs python performance!

[francoislegare]

Hi, I create a model that use 15 tools. I export this model in python. The model takes 35 secondes to run and the python script takes 13 minutes to run. they do the same thing and use the same tools(same geoprocessing tools)  and the result is the same! I tried to modify the script to increase performance but no gains.

Somebody can explane that?
It is possible to increase performance of the script?

I use ArcGis 9.3

Thanks,

SCRIPT:
# ---------------------------------------------------------------------------
# delete_duplicates.py
# Created on: Wed Feb 02 2011 11:21:49 AM
#   (generated by ArcGIS/ModelBuilder)
# Usage: delete_duplicates <block9>
# ---------------------------------------------------------------------------

# Import system modules
import sys, string, os, arcgisscripting

# Create the Geoprocessor object
gp = arcgisscripting.create()
gp.overwriteoutput = 1

# Load required toolboxes...
gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Data Management Tools.tbx")
gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Analysis Tools.tbx")

try:
    # Script arguments...
    fc = string.replace(gp.getparameterastext(0),"\\","/")
    block9 = fc
    if block9 == '#':
        block9 = "block9" # provide a default value if unspecified

    # describe...
    descfc = gp.describe(fc)
    sr = descfc.spatialreference
    dt = descfc.datatype
    cp = descfc.catalogpath
    gp.addmessage("datatype :" + dt)

    #Process the feature class attributes (worckspace)
    lstfc = string.split(cp,"\\")
    #gp.addmessage("split_fc :" + lstfc)
    for fl in lstfc:
        gp.addmessage("split_cp_fl :" + fl)
        fn = fl
    gp.addmessage("fn :" + fn)
    strWorkspace = string.replace(cp,fn,"")
    gp.addmessage("ws :" + strWorkspace)
    gp.workspace = strWorkspace
   
    # Local variables...
    Layer = "Layer"
    doublon_stat_dbf = "C:\\Temp\\doublon_stat.dbf"

    def clearall():
        all = [var for var in globals() if var[0] != " _ "]
        for var in all:
            del globals()[var]


   
  
    if dt in  "ShapeFile, FeatureLayer" :
        gp.addmessage("Traitement des shapes")
        # Process: Add Field...
        gp.addmessage("Add field...")
        gp.AddField_management(fc, "dd", "SHORT", "", "", "", "", "NON_NULLABLE", "NON_REQUIRED", "")

        # Process: Add Field (2)...
        gp.AddField_management(fc, "iddd", "LONG", "", "", "", "", "NON_NULLABLE", "NON_REQUIRED", "")

        # Process: Make Feature Layer...
        gp.addmessage("Make...")
        gp.MakeFeatureLayer_management(fc, Layer)
        #,"", "", "FID_block9 FID_block9 VISIBLE NONE;AREA AREA VISIBLE NONE;PERIMETER PERIMETER VISIBLE NONE;FMFOBJID FMFOBJID VISIBLE NONE;POLYID POLYID VISIBLE NONE;POLYTYPE POLYTYPE VISIBLE NONE;YRSOURCE YRSOURCE VISIBLE NONE;SOURCE SOURCE VISIBLE NONE;FORMOD FORMOD VISIBLE NONE;DEVSTAGE DEVSTAGE VISIBLE NONE;YRDEP YRDEP VISIBLE NONE;OYRORG OYRORG VISIBLE NONE;OSPCOMP OSPCOMP VISIBLE NONE;OLEADSPC OLEADSPC VISIBLE NONE;OAGE OAGE VISIBLE NONE;OHT OHT VISIBLE NONE;OCCLO OCCLO VISIBLE NONE;OSI OSI VISIBLE NONE;OSC OSC VISIBLE NONE;UYRORG UYRORG VISIBLE NONE;USPCOMP USPCOMP VISIBLE NONE;ULEADSPC ULEADSPC VISIBLE NONE;UAGE UAGE VISIBLE NONE;UHT UHT VISIBLE NONE;UCCLO UCCLO VISIBLE NONE;USI USI VISIBLE NONE;USC USC VISIBLE NONE;INCIDSPC INCIDSPC VISIBLE NONE;VERT VERT VISIBLE NONE;HORIZ HORIZ VISIBLE NONE;PRI_ECO PRI_ECO VISIBLE NONE;SEC_ECO SEC_ECO VISIBLE NONE;ACCESS1 ACCESS1 VISIBLE NONE;ACCESS2 ACCESS2 VISIBLE NONE;MGMTCON1 MGMTCON1 VISIBLE NONE;MGMTCON2 MGMTCON2 VISIBLE NONE;MGMTCON3 MGMTCON3 VISIBLE NONE;VERDATE VERDATE VISIBLE NONE;SENSITIV SENSITIV VISIBLE NONE;BED BED VISIBLE NONE;STKG STKG VISIBLE NONE;FID_index_ FID_index_ VISIBLE NONE;OBJECT_ID OBJECT_ID VISIBLE NONE;nom nom VISIBLE NONE;BUFF_DIST BUFF_DIST VISIBLE NONE;dd dd VISIBLE NONE;racc racc VISIBLE NONE;iddd iddd VISIBLE NONE")

        # Process: Calculate Field (5)...
        gp.addmessage("Calculates...")
        gp.CalculateField_management(Layer, "dd", "0", "PYTHON", "")

        # Process: Calculate Field (4)...
        gp.CalculateField_management(Layer, "iddd", "!FID!", "PYTHON", "")

        # Process: Calculate Field...

        expression = "!shape.area@SQUAREMETERS!"

        gp.CalculateField_management(Layer, "AREA", expression, "PYTHON", "")

        # Process: Summary Statistics...
        gp.addmessage("Satistiques...")
        gp.Statistics_analysis(Layer, doublon_stat_dbf, "AREA COUNT;iddd MAX", "AREA")

        # Process: Add Join...
        gp.addmessage("Join...")
        gp.AddJoin_management(Layer, "AREA", doublon_stat_dbf, "AREA", "KEEP_ALL")

        # Process: Select Layer By Attribute...
        gp.addmessage("Select1...")
        gp.SelectLayerByAttribute_management(Layer, "NEW_SELECTION", "\"FREQUENCY\" > 1")

        # Process: Calculate Field (2)...
        gp.CalculateField_management(Layer, "dd", "3", "PYTHON", "")

        # Process: Select Layer By Attribute (2)...
        gp.addmessage("Select2...")
        gp.SelectLayerByAttribute_management(Layer, "NEW_SELECTION", "\"iddd\" = \"MAX_iddd\"")

        # Process: Calculate Field (3)...
        gp.CalculateField_management(Layer, "dd", "4", "PYTHON", "")

        # Process: Select Layer By Attribute (3)...
        gp.addmessage("Select3...")
        gp.SelectLayerByAttribute_management(Layer, "NEW_SELECTION", "\"dd\" =3")

        # Process: Delete Features...
        gp.addmessage("Delete...")
        gp.DeleteFeatures_management(Layer)

        # Process: Delete...
        gp.Delete_management(Layer, "")
    else:
        gp.addmessage("Traitement des autres type non développé")
       

    #Summary message
    gp.addmessage( "features processed")
   
    clearall()  

except:

    gp.addmessage("Error encountered\n")
    gp.getmessage(2)

[OvidioRivero]

I have never been able to make python outperform VB.   I would try exporting the model to VB Script.  Its counter-intuitive because Python is supposed to be more advanced  but VB works better in my opinion.  I suspect its because Python needs to start is own GUI while VB runs natively on Windows.

[RoyceVerboom]

I would love to get my models in VB.NET, but all I see when I'm on Export is "graphic" and "python script".  How do you get it to VB?

I'm running ArcDesktop10.

[NelsonDe_Miranda]

Royce,

Unfortunately that option is no longer available with Arc10 (exporting to VBScript).
This is part of ESRI's deprecation plan of VBScript and focus on Python for geoprocessing.

In 9.3 you used to be able to export to VBScript.

Cheers.

[OvidioRivero]

Yep. That is unfortunate.  I just started taking python training. There is still some support for VBA in 10 but its not worth the effort to use it.  On ESRI's defense I can say that Python in addition to being free is very common among the open source GIS crowd and ESRI is just keeping up with them. Also the autocomplete in the command line works very nice.

[KimOllivier]

The Python code generated by Modelbuilder does not look like an analyst would code the problem. There is huge redundancy and repetition in it that is very inefficient scripting, but is Ok in ModelBuilder.

(If you wrap a <CODE> script... </CODE> pair around your script (the # button on the toolbar) when posting it retains the indentation, otherwise we can't read it.)

You say that you are using 9.3 but I see that the script is using ArcGIS 9.2 .create(), not .create(9.3). Things have moved on with Python now able to run inside the ArcGIS process at 9.3 and 10.0, avoiding creating a separate process for each tool.

Recommendation: Upgrade to 9.3 or 10.0, and run 'in process'. Run it from a tool and check the box 'Run Python script in process' on the Source tab.You may not even need to refactor the code.

But if you do...

System toolboxes do not need to be added, only custom toolboxes not on the search path.

Recommendation: Remove references to system toolboxes to save time and clutter.

CalculateField is designed for ModelBuilder because cursors are not available. Cursors are just as fast and can have a lot more logic built in to do more than one thing on a pass.

Recommendation: Replace multiple CalculateField steps with a single cursor and internal Python logic.

Temporary featureclasses do not need to be written to disk, you can use "in_memory" workspace.
Workspaces should always be a file geodatabase for speed and better indexing, and no size limits.

Recommendation: Don't use dBase tables or shapefiles for intermediate featureclasses.

MakeFeatureLayer does not need to spell out all the fieldmap if you are not modifying it.

Recommendation: Leave the defaults empty if you are not changing it.

Plain try/except blocks are not useful, they hide the error reporting. Remove them and replace them with specific error traps. Never have a plain except: block because it also traps keyboard interrupts. Specify what error you are trying to trap, eg arcgisscripting.ExecuteError, or Exception or something.

Recommendation: Don't use try/except blocks in scripts unless you are actively trapping an error that you want to skip. In a script just let it fail, you will get enough feedback to identify the line and cause. Python errors are rare or even nonexistent since the script is entirely geoprocessing calls.

Python has much faster structures such as lists, dictionaries and sets for finding duplicates. I find it is faster to load selected featureclass attributes once into these python structures, do the comparison and then write out the results, or create an SQL expression to make a layer and write out.

[ChrisFox3]

Also a thing to consider is if your model does everything you want it to, there is no need to export it to python. If for example you want to setup a scheduled task to run your model, you do not need to export it to python you can just call the model from python and pass the necessary paramaters.

As Kim said you are never going to be able to get as efficient a python script just by exporting your model to Python and there are very limited use cases for pursuing this workflow.

gp = arcgisscripting.create(9.3)
gp.AddToolbox("c:/mytoolboxes/custom_tools.tbx")
gp.MyModel_customtools("c:/mytoolboxes/test.gdb/inputFC")