ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

[MarcoBoeringa]

NOTE: This post is NOT about pure Python toolboxes, but ordinary ModelBuilder toolboxes enhanced with Python scripts.

Can ESRI confirm this? I am running into an issue where it seems Pro won't import custom Python modules / scripts in a Calculate Value tool of ModelBuilder. I can successfully import default ArcGIS Python modules like io etc, just not my own modules written as scripts. Pro bails out with a "ModuleNotFoundError".

To avoid confusion, with "import", I mean using a Python module import statement in the Python code textbox of the Calculate Value tools, so like:

import os

import MyModule

This doesn't happen in ArcMap at all. Even if the module is not imported in the toolbox, ArcMap will find it if it is located in the same folder as the toolbox. Pro fails both with a script in the same folder as the toolbox, as well as with imported scripts.

See the attached screenshot from Pro 2.1.2.

[MarcoBoeringa]

I have now tested this:

As is visible, it returns the path of the Pro project folder (which had the default "MyProject1" for this test), where the Project document (*.aprx) is stored.

In ArcMap, it returns the path to the folder where the ArcMap document (*.mxd) is stored, so that is consistent with the Pro result.

However, these paths do not match with the toolbox folder. If I add the toolbox from a location that is clearly not the same as the folder where the project is stored, ArcMap will still function, as the import statements seem interpreted relative to the toolbox location. In Pro, it fails.

[curtvprice]

Are you running the ArcMap tool from ArcToolbox, or from the Catalog window? 

I remember now that a while ago I was dinking with this and saw the same thing you did.   If the tool is added to ArcToolbox I get a different "home" )ie the tbx's folder) than when run from the Catalog window (the ArcMap Home). I bet you may get a third place if run from ArcCatalog's folder tree.  And,  sys.argv[0] and __file__ give me an empty string, not the .tbx path when I call them inside the Calculate Value tool. HOWEVER, I  just thought of a method to meet your use case. 

This possibly would work from ArcMap, ArcCatalog, or Pro, Catalog window or ArcToolbox, if you create it in ArcMap. Please try it and let me know if you get anywhere with my idea.

1. Create a small script tool to find the current folder, place the file in the folder containing tbx and your module.

# findhere.py
import arcpy
import os
arcpy.SetParameterAsText(0, os.path.dirname(os.path.realpath(__file__)))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

2. Add findhere.py script to your .tbx. Go to properties, set relative path checkbox in the first tab. Create a Derived parameter in the parameters tab that is type Folder. Name the parameter Folder.

3. Drop this script tool into your model, try running it to get the path to where findhere.py and your modules you want to import are.

4. Create a Calculate Value tool and make the Folder output a precondition to it.

5. Use this Folder variable in your Calculate Value tool with sys.path.insert(0, path), for example:

# Expression: report()
# Code block:
import sys
sys.path.insert(0, r"%Folder%")
## do your import here ##
def report(x):
  return str(sys.path)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[MarcoBoeringa]

Thanks Curtis.

I will give this a try.

However, your solution of course kind of defeats the whole purpose of the Calculate Value tool, which is to be able to insert and use small Python code snippets inside a geoprocessing model without the need to write a true script and accompanying toolvalidation. If I need to insert a secondary Python script to get the Calculate Value tool to work, I might just as well write it to output the required value the Calculate Value tool was supposed to deliver...

Anyway, from a technical point of view, I am interested to see if your solution works, so will report back on that.

[MarcoBoeringa]

Curtis,

This is really weird. I now attempted to implement your solution, but it still fails.

Actually, you learned me something new, as I wasn't aware you could use the "%variable name%" syntax directly in the Code Block of the Calculate Value tool. I always assumed it could only be used in the Expression text line, and then consumed by a function definition, as that is what the Help shows (and how I implemented it always).

As to the failure:

If you look in this second image, which shows the top of the messaging output related to the Calculate Value tool, you can see that your code actually worked, and correctly inserted the toolbox's / script's folder path ("C:\OSM_Renderer") in the sys.path statement, just before the import of the module I need to access.

Yet, the import still fails!

I am highly confused now . Why does it, if the sys.path is set? It is almost as if Pro ignores sys.path completely, and only uses and searches in the default Pro Python install for modules... I am absolutely 100% sure I got the module name right. Exactly the same code from the same toolbox runs fine in ArcMap.

[MarcoBoeringa]

I am stumped...

I now realized that my toolbox folder should have a space in it:

"C:\OSM Renderer\"

yet the geoprocessing output shows:

"C:\OSM_Renderer\"

so with the space replaced with an underscore...

I now modified your script to see what is going on:

import arcpy
import os

arcpy.AddMessage("BEFORE SetParameterAsText: " + os.path.dirname(os.path.realpath(__file__)))

arcpy.SetParameterAsText(0, os.path.dirname(os.path.realpath(__file__)))

arcpy.AddMessage("AFTER SetParameterAsText: " + arcpy.GetParameterAsText(0))

Now look at the output below, and notice how Pro changed the folder name to contain an underscore instead of space... This is wrong! Either SetParameterAsText or GetParameterAsText mangles the folder path.

[MarcoBoeringa]

If I replace the original folders name with the version with underscore, and re-run the original tool, the module is finally properly imported and the tool runs in Pro! However, this should not happen. Pro should not replace spaces with underscores in arcpy.Get- or SetParamaterAsText function calls.

[MarcoBoeringa]

Curtis,

Well, this settles it. I have now changed the output parameter type of the GetToolboxFolder script that I created using your example code from the suggested Folder to String parameter type.

Pro no longer mangles the output, and the custom Python module I wrote and use in the Calculate Value tool is now found using your suggested code solution. So I have accepted this as the answer with the minor correction that users should set a String instead of Folder parameter type output.

[ShaunWalbridge]

Thanks for the details. Yes, I can see how this inconsistency is annoying. I'm looking into this, and what the ideal behavior is for uses like this. Here are some alternative approaches you can use immediately:

1. Use an explicit path to your external toolbox for the import. This is brittle since it will be hard coded, but it should work fine for your immediate needs.

2. Copy your code into a module, and place that into the Pro site-packages directory (you can create a setuptools package or conda package for this as well). If you're making something more durable, this is a great way to start creating a library around the code you're using, and then you'll be able to import it elsewhere as well.

3. Add a .pth file which links to the directory where your external toolbox is. You can add this to the Pro site-packages, and it will add that location explicitly to the Python search path (for all processes).

[MarcoBoeringa]

Shaun,

Thanks for the suggestions.

However, all of these solutions complicate deployment. With ArcMap, you can just drop the toolbox and scripts it uses anywhere on a file system, and it will import and work properly (assuming scripts and toolbox are in the same folder and scripts have been set to use relative paths, which all of mine are).

I do more and more feel this should be considered a bug, as it breaks current models running perfectly in ArcMap. It also complicates maintaining dual ArcMap / ArcGIS Pro and Python 2/3 compatible toolboxes (which mine is).

As to modules: I actually have written dozens of Python scripts that serve dual roles as geoprocessing tools and modules with functions that I import into other scripts, so as to durability and code maintenance in terms of avoiding code duplication, I think I am OK>

[DanPatterson_Retired]

...  you can just drop the toolbox and scripts  ...

scripts and toolboxes work, just fine in Pro. To clarify, that I haven't misread, your issue is with models and toolboxes