ArcGIS Pro Toolbox & __pycache__

[MErikReedAugusta]

Disclaimer: There seem to be differences here both between ArcMap and ArcPro, as well as Python 2 and Python 3.  That said, the issue seems to be on ArcPro's end.

--------------------------------------------------

Simplified Precis:

I have a geoprocessing tool that runs on a script we'll call Toolscript.py.  This is part of a suite of tools I build & maintain for our team, and they reference a core library that we'll call Assets.py.  Toolscript.py imports Assets.py, and python automatically creates Assets.pyc at runtime.  In the old Desktop days with Python 2, this would be saved in the same place as the Assets.py, whereas Python 3 now puts the compiled file in a __pycache__ subdirectory.

If I make edits to Assets.py, I generally delete Assets.pyc, and have Python generate a fresh one.  Especially if I'm actively writing & testing a script, I will end up doing this frequently, to make sure that my tool that I'm testing is actually using the correct version of the library file.

The Problem:

In ArcGIS Desktop 10.7, deleting Assets.pyc was all I had to do, and I could easily test & iterate on the fly.  My tools always imported the most current version of the library, without restarting ArcGIS.

In ArcGIS Pro 3.1, there seems to be a cached file stored inside ArcGIS Pro.  Even when I delete Assets.pyc, my tools seem to try to reference the old file.  In addition to not getting expected results from whatever fix I just coded, my tracebacks start pointing to nonsense like blank lines (assumedly because the Assets.py and Assets.pyc being referenced by ArcGIS Pro no longer match.

This means any time I make even a minor change to Assets.py, I have to delete Assets.pyc AND close ArcGIS Pro.  This is hugely inefficient when I'm in active R&D and needing to fix things in Assets.py.

Is there a setting I've missed somewhere or some misunderstanding of how ArcGIS Pro handles imported python scripts, now?  I've dug up a few old posts on similar issues, but they mostly seem to have been muddied with other issues and/or not actually solved.  I've also seen reference to importing the reload module from importlib, but I can't help but wonder why I suddenly need this in Pro when I didn't back in Desktop?  I'm not seeing this come up outside of ArcGIS in other Python discussions, though admittedly that searching has been limited.

[DanPatterson · Accepted solution]

related

Solved: Need to restart Pro every time I edit a toolbox's ... - Esri Community

and yes importlib 's reload is the way to go if you import other modules when editing.

It has been too long since using ArcMap's python to know if it was a python 2 vs 3 issue.  

[MErikReedAugusta]

So odd!  This is not a step I ever had to do under ArcGIS Desktop/Python 2, and I was just as regularly tinkering with under-development tools under the hood like this.

Guess I'll have to go dive into the world of importlib, now.

[ShaunWalbridge]

I haven't closely checked, but I think this is due to changes in the caching system of Python 2 vs 3. Deleting the .pyc was a handy trick, but perhaps not a directly supported "feature" from the Python developers perspective, sometimes cached module loading introduces other downstream issues, and typically Python executions expect to only import something once. 

There are tools like IPython's auto reload module which allows hot reloading (https://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html) but using importlib is the best general practice currently because it is explicit. We've discussed having some kind of 'debug' mode that would try to reimport modules, but nothing in the software today for that.

Cheers, Shaun