Does anyone have a good file system hierarchy for their python scripts?

3396
12
02-11-2016 09:09 AM
WesMiller
Regular Contributor III

I have several years of scripts I've written for various projects. I keep them mostly as examples of how to do something so I don't have to figure it out again. I also have scripts i use on a regular basis. Does anyone have a system that they are willing to share on how to store your scripts?

12 Replies
DanPatterson_Retired
MVP Emeritus

My method

  • I make a separate file for all def's as a standalone file ( ie azimuth_py.py, azimuth_np.py) for python and in most cases numpy.
    • rationale 1 ... if I just need a function, I can find it without having to remember the module that it may be in (for example, line_helpers_py.py, line_helpers_np.py)
    • rationale 2 ... a function may be used in several helper modules, and this removes the need to import a piece here and there and the function is already in standalone mode
    • for the above, I keep extensive documentation as to what is where, so if I change a module slightly, I may or may not make the changes in the helper modules.
  • I package def's into helper modules (ie arcpy_py_helper.py, arcpy_np_helper.py (abbrev ANH.py) which contain the helper scripts I need.
  • If I am working on a project, I copy and rename the helper modules to match the project I am working on so I can make changes there without impacting the originals (ie GeomProj_ANH.py which is the arcpy_np_helper.py module used exclusively for the Geometry Project)
  • I back everything up... yes laugh... until you reinvent code for the 3rd time because you haven't followed along so far

That's all I can think of now

0 Kudos
BillDaigle
Occasional Contributor III

Here are some things that work well for us:

  • Use subversion control.  We have 7 gis programmer/analysts and this is the allows us to track changes over time and make updates locally without affecting others.  We use our own internal subversion, but github would work as well.
  • Within our subversion repository, we maintain 5 types of python code:
    • general functions and classes that can be used by other script and toolboxes.  We add the path to this folder to our 'path' environmental variable so we can import modules from it without jumping through too many hoops
    • Scripts.  These are just stand alone python scripts that might be for one time use or a work in progress
    • Python toolboxes: We build most of our data manager and user tools into 'pyt' files so they can be run using the ArcGIS interface.
    • Geoprocessing services:  we store the source code for publish geoprocessing services here
    • Automated tasks: scripts that are scheduled to run on a nightly or weekly basis

Code development generally occurs in standalone scripts or toolboxes.  When we see an opportunity to capitalize on a function or a tool we've developed, we move it into one of the classes in the general functions folder.

Been working great for us!

BrianO_keefe
Occasional Contributor III

If you're looking for a place where you can dump your code snippets, or classes, etc. where the code you are keeping is piecemeal? And not part of a larger package? Then I recommend Evernote. If you use it with PortoCode formatter then you get a nice note that you can add to at anytime.

Example Snippet

http://www.evernote.com/l/AAXZTkmP6pFGzJlQgIJ7gABn_9NcFdz9YQs/

PortoCode Formatter

PortoCode - Portable Codes With Syntax Highlighting On EVERNOTE

Once it's in Evernote then you can search keywords or if you take the time to tag it you get even greater control over accessing the pertinent information at a later date.

0 Kudos
JamesCrandall
MVP Frequent Contributor

TortoiseSVN

We just keep it project-based, much like any other application and components might exist in SVN.

0 Kudos
AdrianWelsh
MVP Honored Contributor

I'll second TortoiseSVN. It's nice because it's free, even for corporate use (open source).

They even have like a tutorial or how-to-use part of their site that helps you get started:

https://tortoisesvn.net/docs/nightly/TortoiseSVN_en/

There are also videos and such on YouTube, like: Tortoise SVN Tutorial - YouTube 

0 Kudos
WesMiller
Regular Contributor III

Thank you everyone for your valuable input!

0 Kudos
PeterWilson
Occasional Contributor III

I've been using Eclipse\PyDev for the last five years and have decided to up my game. I'm currently setting up Python Virtualenv and GitHub to manage my Python Code. I've attached a print screen of my Eclipse\PyDev structure. This allows me to run any of my Python or ArcPy scripts outside of ArcGIS as well as schedule my scripts to run at any given time.

0 Kudos
BillDaigle
Occasional Contributor III

Peter,

Can you elaborate on how this helps you schedule scripts?

Adam Messer

0 Kudos
JamesCrandall
MVP Frequent Contributor

We simply use Windows task scheduler pointing to individual .cmd/.bat files that execute .py scripts.  Aside from managing dev/test/prod versions, it's pretty straight forward.