Change TBX format to a simple ZIP format

[RemigijusPankevičius]

TBX is a proprietary Microsoft OLE2.0 (or as they later named it Compound File Binary File Format). But it is a legacy now; Microsoft had changed from .doc to .docx and .xls to .xslx. These old formats (OLE2.0 based) are legacy now for more than 10 years, while docx and xlsx are a set of XML files compressed into a zip file.

Link: List of Microsoft Office filename extensions - Wikipedia 

For me, representing developers' viewpoint, the legacy is not the main issue.

The biggest issue is that TBX-s is binary format that can't be efficiently used in a development as it exists today. When developer teams are developing different features on different source control (git, tfs to name a few) branches at some point we do merges from several branches into one for the next release the issues with "shared" TBX-es begin. If branch1 got a new tool in common.tbx and branch2 got another one then merge is impossible as TBX is binary format. So we need to solve such conflicts manually. OK, cooperation between the teams is not a bad thing, but...

We can't track differences between "spring release tbx" and "autumn release tbx". We can't see textual diff-s which is essential for finding such issues as what changes could break this version.

So my suggestion is to change to zip with open specification what it should consist: manifest, content, python scripts (and optionally CLSIDS of COM classes, but that's ArcMap world and I do not expect this idea will happen here).

The main wish is to have a set of text files that can be (essential is development with version control) compared, merged, conflicts resolved in text editors. The build step would be a very simple one: zip it.

[DrewFlater]

Hello, I am a lead from the geoprocessing development team and have some information to share.

We are developing a new open-format for toolboxes, based on a folder structure and json files to contain the properties of the toolbox and it's tools and their parameters. This open format will support better source code management, will produce less corruption/invalid binary file problems, and better support backward compatibility across a variety of releases. The story will start to come together in the near-mid term (likely around 2.7).

However, at this time, we can accomplish a lot of what you described specifically for source control by writing your code and toolbox as Python toolboxes. Creating a new Python toolbox—Geoprocessing and Python | ArcGIS Desktop 

This is a single Python file (text file) that contains toolbox and tool definitions as classes. Those classes can come from other modules as well. Python toolboxes are a great capability for developers who want programmatic control of the toolbox and organization of the code using basic Python structures. I hope you can consider this approach. We even have an old utility for auto-generating the PYT based on your TBX: GitHub - Esri/tbx-pyt-translator: Utility for converting a Geoprocessing TBX file to a pure-Python P... .

Lastly, you can explore how some Esri system-tool developers are using Python toolboxes by exploring the PYTs in your ArcGIS Pro installation. C:\Program Files\ArcGIS\Pro\Resources\ArcToolBox\Toolboxes

Drew

[RemigijusPankevičius]

Thanks for sharing it, Drew.

Thumbs up for switching from proprietary binaries to textuality (term by Eric S. Raymond in his book "The Art of Unix Programming").

I've got some inspiration to fill the same idea for changing service definition file (.sd) format.

I've tried Python Toolboxes and quickly started to search for a way how to escape that "quick hack format" rather than a good design decision. It could be better to name Python toolboxes like *.toolbox.py for IDEs to recognize it's a Python file. Also there are issues in using libraries without copy pasting them into .pyt files what is horrible practice.

I found a way how to split that single text file (.pyt) into separate plain Python files (.py). Then .pyt contains only minimal boilerplate code for the toolbox; actual tools are implemented in separate Python files. A "no-op" inheritance is used to put 2 tools in .pyt file. To support refresh-and-continue development import with reload is used.

import arcpy
import AttachmentsTool as attach
import HyperlinksTool as hyperlink

# Explicitly reload modules to recompile (there are issues with that when refreshing .pyt toolbox in ArcMap)
import imp
imp.reload(attach)
imp.reload(hyperlink)

class Toolbox(object):
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the .pyt file)."""
        self.label = "Toolbox"
        self.alias = ""

        # List of tool classes associated with this toolbox
        self.tools = [AttachTool, HyperlinkTool]

class AttachTool(attach.AttachTool):
     pass

class HyperlinkTool(hyperlink.HyperlinkTool):
     pass‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[DanPatterson_Retired]

Although the text representation is welcomed.  The dialog interface for creating custom toolboxes allows for screen grabs to documenting visually, the layout and structure of the toolbox.  Perhaps the visual and text combination that existed when the Dialog Designer was around many years ago would be a nice pairing.

[RemigijusPankevičius]

There is a dialog for documenting tools in toolbox in ArcMap:
A quick tour of documenting tools and toolboxes—Help | ArcGIS for Desktop 

[DanPatterson_Retired]

That I know about

[DrewFlater]

We're marking this implemented with the release of the .atbx format in ArcGIS Pro 2.9. 

See What's new in ArcGIS Pro 2.9