I've been writing Python scripts for a while now, and when a colleague of mine viewed my code, he cringed that it wasn't structured into Python Functions. I'd like some advice from the ESRI community in how I could structure my Python scripts into functions\modules that will make it easier to reuse and call within new scripts. I've attached one of my Python scripts that uses ArcPy and ArcHydro Functions.
'''
Created on May 20, 2015
@author: PeterW
'''
# import system modules and site packages
import os
import arcpy
import ArcHydroTools
# check out Spatial Analyst Extension
arcpy.CheckOutExtension("Spatial")
# set environment settings
arcpy.env.overwriteOutput = True
# set input and output arguments
raw = r"F:\Projects\2015\G111443\ArcHydro\Methodology_Models\Section03\Sect3A\DEM04\raw"
rasWs = r"F:\Projects\2015\G111443\ArcHydro\Methodology_Models\Section03\Sect3A\Layers04"
outWs = r"F:\Projects\2015\G111443\ArcHydro\Methodology_Models\Section03\Sect3A\Model04.gdb"
# ArcHydro variables
fill_sinks = os.path.join(rasWs, "fil")
flow_dir = os.path.join(rasWs, "fdr")
flow_acc = os.path.join(rasWs, "fac")
streams = os.path.join(rasWs, "str")
stream_seg = os.path.join(rasWs, "strlnk")
catchment_grid = os.path.join(rasWs, "cat")
catchment_poly = os.path.join(outWs, "Layers","Catchment")
drainage_line = os.path.join(outWs, "Layers", "DrainageLine")
adj_catch = os.path.join(outWs, "Layers", "AdjointCatchment")
try:
# calculate the fill sinks
arcpy.AddMessage("Processing Fill Sinks")
ArcHydroTools.FillSinks(raw, fill_sinks)
# calculate the flow direction
arcpy.AddMessage("Processing Flow Direction")
ArcHydroTools.FlowDirection(fill_sinks, flow_dir)
# calculate the flow accumulation
arcpy.AddMessage("Processing Flow Accumulation")
ArcHydroTools.FlowAccumulation(flow_dir, flow_acc)
# calculate the maximum flow accumulation
arcpy.AddMessage("Processing Flow Accumulation Maximum")
maxcellsResult = arcpy.GetRasterProperties_management(flow_acc, "MAXIMUM")
maxcells = maxcellsResult.getOutput(0)
print maxcells
# calculate the stream threshold number of cells
arcpy.AddMessage("Processing Stream Threshold")
stream_threshold_numcells = (int(maxcells)*0.25/100)
print stream_threshold_numcells
# calculate the stream definition
arcpy.AddMessage("Processing Stream Definition")
ArcHydroTools.StreamDefinition(flow_acc, stream_threshold_numcells, streams)
# calculate the stream segmentation
arcpy.AddMessage("Processing Stream Segmentation")
ArcHydroTools.StreamSegmentation(streams, flow_dir, stream_seg)
# calculate the catchment grid delineation
arcpy.AddMessage("Processing Catchment Grid Delineation")
ArcHydroTools.CatchmentGridDelineation(flow_dir, stream_seg, catchment_grid)
# calculate the catchment polygons from the catchment grid
arcpy.AddMessage("Processing Catchment Polygons")
ArcHydroTools.CatchmentPolyProcessing(catchment_grid, catchment_poly)
# calculate the drainage lines from the stream segmentation grid
arcpy.AddMessage("Processing DrainageLines")
ArcHydroTools.DrainageLineProcessing(stream_seg, flow_dir, drainage_line)
# calculate the adjoint catchment polygons
arcpy.AddMessage("Processing Ajdoint Catchments")
ArcHydroTools.AdjointCatchment(drainage_line, catchment_poly, adj_catch)
arcpy.AddMessage("Completed Processing ArcHydro Main Model")
except:
print(arcpy.GetMessages(2))
pass
arcpy.CheckInExtension("Spatial")Any advice and assistance will be appreciated.
Regards
Peter Wilson
I'm not one to answer this in full, but I have a few files that I read in each time (thanks to Freddie Gibson and/or Jeff Barrette ) that take care of a few things for me.
These can be place in the (example) c:\Python27\ArcGIS10.3\Lib folder, or in another relative location to your calling script.
the ADFGutils.py (...named for my dept, but you can name whatever you want)
call in your program:
Use similar to print or arcpy.addMessage (may duplicate the message in some debuggers)
import time
import arcpy
import os
from time import localtime
def timeStamp():
"""
returns time stamp.
"""
return time.strftime(' -- %B %d - %H:%M:%S')
def myMsgs(message):
arcpy.AddMessage(message + ' %s' %(timeStamp()))
print(message + ' %s' %(timeStamp()))
global messageCount
logFolder = r"C:\ESRITEST"
if not arcpy.Exists(logFolder):
arcpy.CreateFolder_management(os.sep.join(logFolder.split(os.sep)[:-1]), logFolder.split(os.sep)[-1])
mdy = curDate()
logName = "logfile_" + "_".join(mdy.split("/")) + ".log"
logFile = open(os.path.join(logFolder, logName), "a") #a=append, w=create new
if message.lower() == "blank line":
logFile.write("\n\n")
print "\n\n"
elif message.lower() == "close logfile":
logFile.write("\n\n***** finished *****\n\n")
logFile.close()
else:
messageCount += 1
logFile.write("0" * (5 - len(str(messageCount))) + str(messageCount) + ". ")
logFile.write(message)
logFile.write("\n")
#print message
#arcpy.AddMessage(message)
def curDate():
rawTime = localtime()
yr = str(rawTime[0]) # Collect the year from the rawTime variable
mo = str(rawTime[1]) # Collect the month from the rawTime variable
dy = str(rawTime[2]) # Collect the day from the rawTime variable
return "/".join([mo, dy, yr])
messageCount = 0
The gpdecorator.py
To call in your program
To use....at the
# catch_errors decorator must preceed a function using the @ notation.
@catch_errors
def main():
"""
Main function to create the new master feature dataset.
"""
# Script arguments...
'''.......your program'''
| myMsgs('!!! Success !!! ') |
# End main function
if __name__ == '__main__':
main()
"""
A decorator to wrap error handling.
"""
import sys as _sys
import traceback as _traceback
import arcpy
def catch_errors(func):
"""
Decorator function to support error handling
"""
def decorator(*args, **kwargs):
"""
Decorator function
"""
try:
f = func(*args, **kwargs)
return f
except Exception:
tb = _sys.exc_info()[2]
tbInfo = _traceback.format_tb(tb)[-1]
arcpy.AddError('PYTHON ERRORS:\n%s\n%s: %s\n' %
(tbInfo, _sys.exc_type, _sys.exc_value))
print('PYTHON ERRORS:\n%s\n%s: %s\n' %
(tbInfo, _sys.exc_type, _sys.exc_value))
gp_errors = arcpy.GetMessages(2)
if gp_errors:
arcpy.AddError('GP ERRORS:\n%s\n' % gp_errors)
print('GP ERRORS:\n%s\n' % gp_errors)
# End decorator function
return decorator
# End catch_errors function
if __name__ == '__main__':
passThat may not be all your colleague was pointing out, but I know those help me. These most likely could be in one file, btw, but I've never combined them. fwiw.
For me, the main reasons to refactor code into functions (and classes, modules, packages...) are readability and reusability.
On readability - if I have a long script with a number of steps, each with a number of lines of code, I will usually bust each step out into a function so it's easier to follow the overall logic of the script.
On reusability - if I find myself writing the same code over and over again to do the same thing with different inputs, I will turn that code into a function.
I don't see anything in your code that requires moving into a function. It's a very straightforward script, is very readable and there's nothing in it that is reusable.
A cautionary note... it's easy to get too caught up in refactoring and go too far. I was recently reviewing another programmers code where everything was a function, and functions called functions which called more functions ad-infinitum and understanding what the script was doing meant jumping all over the place in the file. It was basically unreadable.
Hi Luke
Thanks for you advice, it's truly appreciated. Is there anyway of better handling my Arc Hydro Variables. The inputs\outputs are either rasters or feature classes\tables and they are being read\written to two workspaces.
The rasters are being saved or read from a single folder and the feature classes\tables are being saved or read from a File Geodatabase.
Regards
Peter Wilson
Peter, your variables look fine to me. The only suggestion I might make is to pass your inputs in to the script using arguments and access them with sys.argv or arcpy.GetParameterAsText. And I'd only recommend that if you're going to be using the code as a commandline or ArcToolbox script tool and you may want to use different inputs/workspaces when running the code.
I write code that ranges from simple step-by-step scripts to large libraries that are spread across multiple files in python packages. Here's how I roughly think about when deciding whether to modularise/refactor code:
I would like to agree with your cautionary note. Also, regarding OOP, I always keep this in mind:
The problem with object-oriented languages is they've got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.
-Joe Armstrong, developer of Erlang programming language.
I like OOP, I do, but sometimes I've seen cases where this has definitely been true.
I don't have any comments regarding the formatting/structure of the code. I think others have already provided good feedback on that topic.
In taking a quick look over your code, I do have a couple questions regarding content, if you are open to discussing content as well as structure.