Python cache for custom toolbox: why?

[JackKelley]

ArcGIS Pro : Python cache for custom toolbox

ArcGIS Desktop versus ArcGIS Pro : custom toolbox

I am writing Python scripts and modules for a toolbox. Everything (toolbox, scripts, modules) is kept in one folder ("Toolbox Version 1"). The tool scripts import from the modules (e.g. "import helper"). Some modules import from others in the same folder.

To make a new version of the toolbox, I can simply copy the folder as "Toolbox Version 2" and work there. The scripts are enhanced and the <helper> module is revised. The Python files are revised, but not renamed.

ArcGIS Desktop (10.6.1) executes the tools in both toolboxes promptly and correctly.

ArcGIS Pro : ???????

In ArcGIS Pro 2.2.4, a toolbox's first-time use involves a lengthy wait (nearly a minute) before starting (for even the simplest script). Presumably, Pro is caching all of the Python for the project.

New project. Add Toolbox Version 1. Click on a script tool. Wait 45 seconds while the unexplained activity proceeds. Run the tools ... Time passes ... Add Toolbox Version 2. Will the new <helper> module be used? No, the old cached <helper> is used. Will removing Toolbox Version 1 do the trick? No. You will have to remove both toolboxes, save the project, and exit ArcGIS Pro.

WHY ???

See also https://community.esri.com/message/808600-arcgispro-python-toolbox-problem-with-modulepy-cache-copy-...

[DanPatterson_Retired]

It is a python 3 thing 

some links on Stack in this thread

python - What is __pycache__? - Stack Overflow 

There are links on how to disable it if it is a real issue for you

I just delete the __pycache__ folder(s) in my module path before moving to a new location.  An inconvenience but not one that can't be overcome

[JackKelley]

At the very least, I want ArcGIS Pro's toolbox Refresh operation to completely refresh the toolbox.

Please try the attached test, to see how ArcGIS Pro can "misunderstand" the __pycache__ for a toolbox.

You will find version1 and version2 folders, each containing the same toolbox. Neither folder has a __pycache__ (to begin with).

Suppose you are working in a single project. If you add both toolboxes, you can observe behaviour like this:

Add version1 toolbox (no __pycache__)
Add version2 toolbox (no __pycache__)

Run Script in version2 (delay of about 50 seconds)

Start Time: Thursday, 16 May 2019 11:28:23 AM
Running script Script...
<helper> version 2 :with: <helper_helper> version 2
Script version 2
Completed script Script...
Succeeded at Thursday, 16 May 2019 11:28:28 AM (Elapsed Time: 4.37 seconds)

__pycache__ then appears in version2
no __pycache__ in version1

Run Script in version1 (no delay)

Start Time: Thursday, 16 May 2019 11:39:41 AM
Running script Script...
<helper> version 2 :with: <helper_helper> version 2
Script version 1
Completed script Script...
Succeeded at Thursday, 16 May 2019 11:39:41 AM (Elapsed Time: 0.03 seconds)

no __pycache__ in version1

Delete __pycache__ in version2

Run Script in version1 (no delay)

Start Time: Thursday, 16 May 2019 11:41:38 AM
Running script Script...
<helper> version 2 :with: <helper_helper> version 2
Script version 1
Completed script Script...
Succeeded at Thursday, 16 May 2019 11:41:38 AM (Elapsed Time: 0.03 seconds)

Run RefreshToolbox in version1   (this reloads the imported modules)

Start Time: Thursday, 16 May 2019 11:45:23 AM
Running script RefreshToolbox...
<helper_helper> version 2
<helper_helper> version 1
<helper> version 2 :with: <helper_helper> version 2
<helper> version 1 :with: <helper_helper> version 1
Completed script RefreshToolbox...
Succeeded at Thursday, 16 May 2019 11:45:23 AM (Elapsed Time: 0.07 seconds)

__pycache__ then appears in version1

Run Script in version1 (no delay)

Start Time: Thursday, 16 May 2019 11:46:30 AM
Running script Script...
<helper> version 1 :with: <helper_helper> version 1
Script version 1
Completed script Script...
Succeeded at Thursday, 16 May 2019 11:46:30 AM (Elapsed Time: 0.03 seconds)

[DanPatterson_Retired]

Are you editing one or more of the scripts during this process Jack? There are too many variables in importlib to make sense of what will happen if any of the python objects are retained or completely removed before using reload from importlib.

importlib — The implementation of import — Python 3.7.3 documentation 

[JackKelley]

Dan, I understand that the reloads (in RefreshToolbox using "Refresh.py", which I should not have to write) are potentially risky. However, in this case the only difference between the versions is the version number.

"helper_helper.py" (version1)

version_string = "<helper_helper> version 1"

"helper.py" (version1)

version_string = "<helper> version 1"
import helper_helper
version_string = "%s :with: %s" % (version_string, helper_helper.version_string)

"Script.py" (version1)

import arcpy
import helper

def message (text):
   arcpy.AddMessage (text)
   print (arcpy.GetMessages ())

message (helper.version_string)
message ("Script version 1")

"Refresh.py" (version1) and (version2)

import arcpy

def message (text):
   arcpy.AddMessage (text)
   print (arcpy.GetMessages ())

from importlib import reload

import helper_helper
message (helper_helper.version_string)
reload (helper_helper)
message (helper_helper.version_string)

import helper
message (helper.version_string)
reload (helper)
message (helper.version_string)

I have used "Store tool with relative path" for all tools.

As you can see from my previous log of observations, ArcGIS Pro does not always look for the __pycache__ in the correct folder. And worse (see 3rd last item in the log, after deleting version2 __pycache__).

The same bad behaviour seems to occur (within a single project) whether you add a toolbox to the project or simply find it by browsing to its folder. And as noted in the original question, the toolbox Remove operation does not remove all traces of it.

The toolbox right-click Refresh operation merely says "Update the contents list to include any items that were recently added to this location".

I think that ArcGIS Pro's toolbox Refresh operation should completely refresh the toolbox, that is, completely revise its conception of where each (source) part of that toolbox is (stored as an operating-system file).