topic Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder in ArcGIS Pro Questions

ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

MarcoBoeringa — Thu, 29 Mar 2018 21:23:11 GMT

NOTE: This post is NOT about pure Python toolboxes, but ordinary ModelBuilder toolboxes enhanced with Python scripts.

Can ESRI confirm this? I am running into an issue where it seems Pro won't import custom Python modules / scripts in a Calculate Value tool of ModelBuilder. I can successfully import default ArcGIS Python modules like io etc, just not my own modules written as scripts. Pro bails out with a "ModuleNotFoundError".

To avoid confusion, with "import", I mean using a Python module import statement in the Python code textbox of the Calculate Value tools, so like:

import os

import MyModule

This doesn't happen in ArcMap at all. Even if the module is not imported in the toolbox, ArcMap will find it if it is located in the same folder as the toolbox. Pro fails both with a script in the same folder as the toolbox, as well as with imported scripts.

See the attached screenshot from Pro 2.1.2.

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

DanPatterson_Retired — Thu, 29 Mar 2018 22:00:09 GMT

Marco... perhaps models are set up differently. I import my own modules (aka scripts) into my scripts in conventional toolboxes all the time, where they are found if they are located in the same location as the toolbox's script.

Export your model to a script to see if the paths are hardcoded or not

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

MarcoBoeringa — Thu, 29 Mar 2018 22:18:37 GMT

Dan,

This is not about importing scripts / modules into another script / module part of a conventional ModelBuilder toolbox. Like you say, that works both in ArcMap and Pro.

The problem is specifically with custom scripts / modules imported with the Python import statement inside the code block of a Calculate Value "ModelBuilder-only-tool". That works in ArcMap, but fails in Pro if the imported module is a custom script I wrote myself, so not part of the default ArcGIS Python install.

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

DanPatterson_Retired — Thu, 29 Mar 2018 22:31:06 GMT

Ok... so it doesn't work the same way. there is too much % this % that stuff in the help topic to figure out and there is no mention of imports.... sorry

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

DanPatterson_Retired — Fri, 30 Mar 2018 00:32:54 GMT

curtvprice‌ you use modelbuilder. in Pro?

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

curtvprice — Sat, 11 Dec 2021 09:49:24 GMT

I made a little test model tool to look at this. I created it in ArcMap and created it in Pro. Comparing the results, I found something very interesting, that you can see from the Python prompts. Pro's sys.path does not have "" as the first element as we normally see in most Python implementations!

Note, my experience is that os.curdir (which is what the blank entry means in the sys.path) is not the folder with the tbx, but the current ArcMap home folder (where the mxd is saved) or the Pro project folder. Would like to know a good way to do your use case (find the current folder where the model's tbx lives) no matter where the tbx is relative to the map document or project.

ArcMap 10.5's Python window:

>>> import sys
>>> sys.path[0]
''‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

From Pro's Python window:

import sys
sys.path[0]
'c:\\program files\\arcgis\\pro\\Resources\\arcpy'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I think you can get your script to work by adding the current folder to the sys.path. Maybe.

# Calculate value expression:  f()
# Code block:
import sys
sys.path.insert(0, "")
import ### your module ###
def f():
  # do stuff
  return ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

My guess is this is a feature not a bug. Shaun Walbridge‌ do you have any enlightenment for us?

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

curtvprice — Fri, 30 Mar 2018 05:06:34 GMT

Can ESRI confirm this?

Just a note: if you want to report a bug, you need to open a support incident with support.esri.com -- this is a community site which is useful, but not the place to report bugs.

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

ShaunWalbridge — Sat, 11 Dec 2021 09:49:28 GMT

Curtis Price‌ Thanks for digging into this. I'm doing some research on my end, and will post here when that's done. For now, I recommend doing:

import sys
sys.path.insert(0, '')

In the code block, as you mentioned in your second example. This is the exact behavior of a typical Python process, and will prepend the working directory of the current Pro project to the search path for Python code. There are valid reasons to block this behavior in an embedded use, but I'm checking if they apply to Pro specifically.

Cheers,

Shaun

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

MarcoBoeringa — Fri, 30 Mar 2018 22:59:19 GMT

Thanks Curtis for all the detective work. Yes, I know about the support stuff, but as I wasn't exactly sure if this was a "bug" or, as you stated, a possible "feature", so I considered posting here more appropriate for now.

In addition, since this seemed to be a kind of trivial issue related to path interpretation, I hoped someone in the community like you, would have ideas for a solution (as you did).

Unfortunately, despite implementing your solution in the top part of the code block of the Calculate Value tool, it still fails to locate the module in Pro, and errors out with the ModuleNotFoundError, as I showed in the screenshot of the first post.

As I also wrote there, the same model runs fine in ArcMap.

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

curtvprice — Sat, 11 Dec 2021 09:49:31 GMT

I'm thinking that in Pro the the home folder may not where you think it is. In your model, try setting up a Calculate Value tool that shows this path will help you check, below.

For your import to work the returned path will have to be the folder where the module you are trying to import lives.

# Calculate Value expression: test()
# Code block:
import os
def test():
    return os.path.realpath(os.curdir)‍‍‍‍‍‍‍‍‍‍

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

MarcoBoeringa — Fri, 30 Mar 2018 23:56:36 GMT

I have now tested this:

As is visible, it returns the path of the Pro project folder (which had the default "MyProject1" for this test), where the Project document (*.aprx) is stored.

In ArcMap, it returns the path to the folder where the ArcMap document (*.mxd) is stored, so that is consistent with the Pro result.

However, these paths do not match with the toolbox folder. If I add the toolbox from a location that is clearly not the same as the folder where the project is stored, ArcMap will still function, as the import statements seem interpreted relative to the toolbox location. In Pro, it fails.

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

curtvprice — Sat, 11 Dec 2021 09:49:33 GMT

Are you running the ArcMap tool from ArcToolbox, or from the Catalog window?

I remember now that a while ago I was dinking with this and saw the same thing you did. If the tool is added to ArcToolbox I get a different "home" )ie the tbx's folder) than when run from the Catalog window (the ArcMap Home). I bet you may get a third place if run from ArcCatalog's folder tree. And, sys.argv[0] and __file__ give me an empty string, not the .tbx path when I call them inside the Calculate Value tool. HOWEVER, I just thought of a method to meet your use case.

This possibly would work from ArcMap, ArcCatalog, or Pro, Catalog window or ArcToolbox, if you create it in ArcMap. Please try it and let me know if you get anywhere with my idea.

1. Create a small script tool to find the current folder, place the file in the folder containing tbx and your module.

# findhere.py
import arcpy
import os
arcpy.SetParameterAsText(0, os.path.dirname(os.path.realpath(__file__)))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

2. Add findhere.py script to your .tbx. Go to properties, set relative path checkbox in the first tab. Create a Derived parameter in the parameters tab that is type Folder. Name the parameter Folder.

3. Drop this script tool into your model, try running it to get the path to where findhere.py and your modules you want to import are.

4. Create a Calculate Value tool and make the Folder output a precondition to it.

5. Use this Folder variable in your Calculate Value tool with sys.path.insert(0, path), for example:

# Expression: report()
# Code block:
import sys
sys.path.insert(0, r"%Folder%")
## do your import here ##
def report(x):
  return str(sys.path)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

ShaunWalbridge — Sat, 31 Mar 2018 04:39:19 GMT

Thanks for the details. Yes, I can see how this inconsistency is annoying. I'm looking into this, and what the ideal behavior is for uses like this. Here are some alternative approaches you can use immediately:

1. Use an explicit path to your external toolbox for the import. This is brittle since it will be hard coded, but it should work fine for your immediate needs.

2. Copy your code into a module, and place that into the Pro site-packages directory (you can create a setuptools package or conda package for this as well). If you're making something more durable, this is a great way to start creating a library around the code you're using, and then you'll be able to import it elsewhere as well.

3. Add a .pth file which links to the directory where your external toolbox is. You can add this to the Pro site-packages, and it will add that location explicitly to the Python search path (for all processes).

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

MarcoBoeringa — Sat, 31 Mar 2018 08:23:19 GMT

Thanks Curtis.

I will give this a try.

However, your solution of course kind of defeats the whole purpose of the Calculate Value tool, which is to be able to insert and use small Python code snippets inside a geoprocessing model without the need to write a true script and accompanying toolvalidation. If I need to insert a secondary Python script to get the Calculate Value tool to work, I might just as well write it to output the required value the Calculate Value tool was supposed to deliver...

Anyway, from a technical point of view, I am interested to see if your solution works, so will report back on that.

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

MarcoBoeringa — Sat, 31 Mar 2018 08:29:43 GMT

Shaun,

Thanks for the suggestions.

However, all of these solutions complicate deployment. With ArcMap, you can just drop the toolbox and scripts it uses anywhere on a file system, and it will import and work properly (assuming scripts and toolbox are in the same folder and scripts have been set to use relative paths, which all of mine are).

I do more and more feel this should be considered a bug, as it breaks current models running perfectly in ArcMap. It also complicates maintaining dual ArcMap / ArcGIS Pro and Python 2/3 compatible toolboxes (which mine is).

As to modules: I actually have written dozens of Python scripts that serve dual roles as geoprocessing tools and modules with functions that I import into other scripts, so as to durability and code maintenance in terms of avoiding code duplication, I think I am OK>

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

MarcoBoeringa — Sat, 31 Mar 2018 09:36:25 GMT

Curtis,

This is really weird. I now attempted to implement your solution, but it still fails.

Actually, you learned me something new, as I wasn't aware you could use the "%variable name%" syntax directly in the Code Block of the Calculate Value tool. I always assumed it could only be used in the Expression text line, and then consumed by a function definition, as that is what the Help shows (and how I implemented it always).

As to the failure:

If you look in this second image, which shows the top of the messaging output related to the Calculate Value tool, you can see that your code actually worked, and correctly inserted the toolbox's / script's folder path ("C:\OSM_Renderer") in the sys.path statement, just before the import of the module I need to access.

Yet, the import still fails!

I am highly confused now . Why does it, if the sys.path is set? It is almost as if Pro ignores sys.path completely, and only uses and searches in the default Pro Python install for modules... I am absolutely 100% sure I got the module name right. Exactly the same code from the same toolbox runs fine in ArcMap.

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

DanPatterson_Retired — Sat, 31 Mar 2018 09:38:14 GMT

... you can just drop the toolbox and scripts ...

scripts and toolboxes work, just fine in Pro. To clarify, that I haven't misread, your issue is with models and toolboxes

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

MarcoBoeringa — Sat, 31 Mar 2018 11:37:23 GMT

I am stumped...

I now realized that my toolbox folder should have a space in it:

"C:\OSM Renderer\"

yet the geoprocessing output shows:

"C:\OSM_Renderer\"

so with the space replaced with an underscore...

I now modified your script to see what is going on:

import arcpy
import os

arcpy.AddMessage("BEFORE SetParameterAsText: " + os.path.dirname(os.path.realpath(__file__)))

arcpy.SetParameterAsText(0, os.path.dirname(os.path.realpath(__file__)))

arcpy.AddMessage("AFTER SetParameterAsText: " + arcpy.GetParameterAsText(0))

Now look at the output below, and notice how Pro changed the folder name to contain an underscore instead of space... This is wrong! Either SetParameterAsText or GetParameterAsText mangles the folder path.

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

MarcoBoeringa — Sat, 31 Mar 2018 11:41:21 GMT

If I replace the original folders name with the version with underscore, and re-run the original tool, the module is finally properly imported and the tool runs in Pro! However, this should not happen. Pro should not replace spaces with underscores in arcpy.Get- or SetParamaterAsText function calls.

Re: ArcGIS Pro bug?: Can't import custom Python module in "Calculate Value" tool of ModelBuilder

MarcoBoeringa — Sat, 31 Mar 2018 12:03:57 GMT

Curtis,

Well, this settles it. I have now changed the output parameter type of the GetToolboxFolder script that I created using your example code from the suggested Folder to String parameter type.

Pro no longer mangles the output, and the custom Python module I wrote and use in the Calculate Value tool is now found using your suggested code solution. So I have accepted this as the answer with the minor correction that users should set a String instead of Folder parameter type output.