topic Re: Does anyone have a good file system hierarchy for their python scripts? in Python Questions

Does anyone have a good file system hierarchy for their python scripts?

WesMiller — Thu, 11 Feb 2016 17:09:56 GMT

I have several years of scripts I've written for various projects. I keep them mostly as examples of how to do something so I don't have to figure it out again. I also have scripts i use on a regular basis. Does anyone have a system that they are willing to share on how to store your scripts?

Re: Does anyone have a good file system hierarchy for their python scripts?

DanPatterson_Retired — Thu, 11 Feb 2016 17:28:40 GMT

My method

I make a separate file for all def's as a standalone file ( ie azimuth_py.py, azimuth_np.py) for python and in most cases numpy.
- rationale 1 ... if I just need a function, I can find it without having to remember the module that it may be in (for example, line_helpers_py.py, line_helpers_np.py)
- rationale 2 ... a function may be used in several helper modules, and this removes the need to import a piece here and there and the function is already in standalone mode
- for the above, I keep extensive documentation as to what is where, so if I change a module slightly, I may or may not make the changes in the helper modules.
I package def's into helper modules (ie arcpy_py_helper.py, arcpy_np_helper.py (abbrev ANH.py) which contain the helper scripts I need.
If I am working on a project, I copy and rename the helper modules to match the project I am working on so I can make changes there without impacting the originals (ie GeomProj_ANH.py which is the arcpy_np_helper.py module used exclusively for the Geometry Project)
I back everything up... yes laugh... until you reinvent code for the 3rd time because you haven't followed along so far

That's all I can think of now

Re: Does anyone have a good file system hierarchy for their python scripts?

BillDaigle — Thu, 11 Feb 2016 18:29:07 GMT

Here are some things that work well for us:

Use subversion control. We have 7 gis programmer/analysts and this is the allows us to track changes over time and make updates locally without affecting others. We use our own internal subversion, but github would work as well.
Within our subversion repository, we maintain 5 types of python code:
- general functions and classes that can be used by other script and toolboxes. We add the path to this folder to our 'path' environmental variable so we can import modules from it without jumping through too many hoops
- Scripts. These are just stand alone python scripts that might be for one time use or a work in progress
- Python toolboxes: We build most of our data manager and user tools into 'pyt' files so they can be run using the ArcGIS interface.
- Geoprocessing services: we store the source code for publish geoprocessing services here
- Automated tasks: scripts that are scheduled to run on a nightly or weekly basis

Code development generally occurs in standalone scripts or toolboxes. When we see an opportunity to capitalize on a function or a tool we've developed, we move it into one of the classes in the general functions folder.

Been working great for us!

Re: Does anyone have a good file system hierarchy for their python scripts?

BrianO_keefe — Thu, 11 Feb 2016 18:45:46 GMT

If you're looking for a place where you can dump your code snippets, or classes, etc. where the code you are keeping is piecemeal? And not part of a larger package? Then I recommend Evernote. If you use it with PortoCode formatter then you get a nice note that you can add to at anytime.

Example Snippet

http://www.evernote.com/l/AAXZTkmP6pFGzJlQgIJ7gABn_9NcFdz9YQs/

PortoCode Formatter

PortoCode - Portable Codes With Syntax Highlighting On EVERNOTE

Once it's in Evernote then you can search keywords or if you take the time to tag it you get even greater control over accessing the pertinent information at a later date.

Re: Does anyone have a good file system hierarchy for their python scripts?

JamesCrandall — Thu, 11 Feb 2016 19:23:12 GMT

TortoiseSVN

We just keep it project-based, much like any other application and components might exist in SVN.

Re: Does anyone have a good file system hierarchy for their python scripts?

AdrianWelsh — Thu, 11 Feb 2016 23:00:44 GMT

I'll second TortoiseSVN. It's nice because it's free, even for corporate use (open source).

They even have like a tutorial or how-to-use part of their site that helps you get started:

https://tortoisesvn.net/docs/nightly/TortoiseSVN_en/

There are also videos and such on YouTube, like: Tortoise SVN Tutorial - YouTube

Re: Does anyone have a good file system hierarchy for their python scripts?

WesMiller — Tue, 16 Feb 2016 14:43:19 GMT

Thank you everyone for your valuable input!

Re: Does anyone have a good file system hierarchy for their python scripts?

PeterWilson — Wed, 17 Feb 2016 20:38:20 GMT

I've been using Eclipse\PyDev for the last five years and have decided to up my game. I'm currently setting up Python Virtualenv and GitHub to manage my Python Code. I've attached a print screen of my Eclipse\PyDev structure. This allows me to run any of my Python or ArcPy scripts outside of ArcGIS as well as schedule my scripts to run at any given time.

Re: Does anyone have a good file system hierarchy for their python scripts?

BillDaigle — Wed, 17 Feb 2016 20:45:24 GMT

Peter,

Can you elaborate on how this helps you schedule scripts?

Adam Messer

Re: Does anyone have a good file system hierarchy for their python scripts?

JamesCrandall — Wed, 17 Feb 2016 21:47:33 GMT

We simply use Windows task scheduler pointing to individual .cmd/.bat files that execute .py scripts. Aside from managing dev/test/prod versions, it's pretty straight forward.

Re: Does anyone have a good file system hierarchy for their python scripts?

PeterWilson — Wed, 17 Feb 2016 22:34:19 GMT

Hi Bill

The following article might help you.

Scheduling a Python script.

Re: Does anyone have a good file system hierarchy for their python scripts?

BillDaigle — Wed, 17 Feb 2016 23:13:36 GMT

Thanks Peter,

We already use window's task scheduler. It works OK, but is a pain whenever I leave the office and take my computer with me. We end up manually setting up the tasks on another user's computer temporarily, then disabling them when I get back in the office. Long term, I think we're better off running the tasks on a server instead of my desktop.

I have not used virtualenv, so I was curious if this was giving you more flexibility when implementing scheduled tasks.

Re: Does anyone have a good file system hierarchy for their python scripts?

WesMiller — Wed, 17 Feb 2016 23:33:59 GMT

The server is where i run my scripts with task scheduler.