Python Tool Management System

[HaydenWelch]

edit:

I've uploaded the basic Modular Toolbox framework to GitHub

I've been using Python Toolboxes in production workflows for a while now and have been really irritated with how difficult it is to maintain toolsets that need to be branched between projects.

The whole system is a bit difficult to use with all your tools for a toolbox being in a single pyt file.

I've built a basic system for expanding this functionality using Python's module system and am wondering if there's any interest in the community for me open sourcing the basic framework.

There are some weird bugs still, Arc doesn't reload modules when you refresh a toolbox, so module tools that are updated during production won't get changes until a full Arc reboot. I got around this with explicit reload calls for each tool module in the pyt.

So far it's pretty basic, with a module structure that has toolboxes in the root directory and tools stored in project sub modules within a tool module. (e.g. `tools.<project>.<tool_name>.<tool_class_name>).

There's also a wrapper class for tools that allows you to inherit properties and methods from this base class so you aren't re-implementing the same functions and parameter definitions between tools. So your tools will be defined as `class ToolName(ToolWrapper)` instead of base objects.

So far it's been working well for us and if there's interest from you guys I'd be willing to open source the base framework so other people can start building toolboxes with it.

[BlakeTerhune]

I think there are ways to make a modular toolbox that reloads each time. For example, here's the main toolbox file that imports tools from separate .py files. I assume they're stored in a folder called "scripts" in the same directory as the main toolbox file.

import os
import sys

# Import custom script tools.
ff_tools_rel_path = os.path.join(os.path.dirname(__file__), "scripts")
sys.path.append(ff_tools_rel_path)
import demo_tool

# Reload the custom scripts to allow ArcGIS Pro to see changes on refresh.
# This prevents having to restart ArcGIS Pro when changes are made.
from importlib import reload
reload(demo_tool)
from demo_tool import demoTool

class Toolbox(object):
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of
        the .pyt file)."""
        self.label = "Parent Toolbox Demo"
        self.alias = "DemoToolbox"

        # List of tool classes associated with this toolbox
        self.tools = [
            demoTool
        ]

Here's an example of the .py file for a tool.

import arcpy
import helper_library as helper

import importlib
importlib.reload(helper)

class demoTool(object):
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "Demo Tool"
        self.description = ""
        self.canRunInBackground = False

    def getParameterInfo(self):
        """Define parameter definitions"""
        params = None
        return params

    def isLicensed(self):
        """Set whether tool is licensed to execute."""
        return True

    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        return

    def updateMessages(self, parameters):
        """Modify the messages created by internal validation for each tool
        parameter.  This method is called after internal validation."""
        return

    def execute(self, parameters, messages):
        """The source code of the tool."""
        arcpy.AddMessage(helper.exampleFunction())
        return

You'll notice it also imports a "helper_library" file. It's an example of having another .py file acting like a custom library for helper functions that might be shared across tools. The helper function used in the example above would be in a simple .py file too.

def exampleFunction():
    return "I can help!"

[BlakeTerhune]

I've only made one attempt at this though. I'd still like to see your framework. Perhaps Python Documents would be a better place for this discussion though?

[HaydenWelch]

@BlakeTerhuneThis is my first post on the forums, sorry for putting it in the wrong spot. I'll clean it up and re-post this discussion there tomorrow.

Also your example code is basically exactly how my current toolbox file is structured. I have a separate module for holding helper submods and constants and a module for the tools with submodules for their projects. The only addition I have is a fallback tool class that a tool that fails to import turns into that writes out the stack trace to the description and puts it in a separate "In Development" category

I can't seem to post anything in the Documents forum. So I'lll just update this post here.

Here's the repo for the base framework: https://github.com/hwelch-fle/pytframe

[HaydenWelch]

@BlakeTerhune 

Updated the post with links to the repo. It's still pretty rough, but as far as I know this is the best way to implement something like this currently. Makes it easier to manage toolboxes that are in active development with the fallback class and breaks each tool into an individually trackable file that's essentially plug and place of you need to rewrite it to work in another context.

Since we work with a lot of projects that have different requirements, breaking out the process like this makes building a project specific toolbox way easier as I can just create a project submodule and import the modified tool from there without having to extend or generalize the tool that's already in production.

The helper module also encourages people to find general solutions to parts of their problems and share them so anyone else working on a toolbox can use them without having to copy and paste and any tools using those general helper functions get free upgrades when they're updated.

For more context we also use toolboxes from a central location so everyone is using the same file. Which is why I built in the fallback. We were originally copying the toolbox into each project, but that ended up causing issues when tools were updated and there were multiple copies floating around that had different behavior.

[Brian_Wilson]

My attempt at solving these problems is here: https://github.com/Wildsong/ArcGIS_Python_Template I've been pecking away at it for years. Feel free to borrow or steal ideas for your framework.

I look forward to checking out your repo later today.

[HaydenWelch]

@Brian_WilsonIt's not stealing if everyone's sharing! Thanks for the link, I'll definitely try and roll some of your solutions into mine. I'll add you as a collaborator on my repo too if you want to work together on this problem

[Luke_Pinner]

Nice work. Might need to change your import syntax though if you want to make the toolboxes publishable as GP Services. I think this might still be an issue.

https://community.esri.com/t5/arcgis-pro-questions/not-all-scripts-copy-to-server-when-publishing-gp...

[HaydenWelch]

Thanks for the suggestion. I haven't published anything yet so haven't run into that issue. I'll add a patch at some point or at least document the issue for others who do need to publish.