topic Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure) in ArcGIS Pro Questions

Import custom arcpy modules in Pro 1.4 (from Python addin structure)

RebeccaStrauch__GISP — Wed, 19 Apr 2017 00:55:19 GMT

I'm trying to test some of my custom tools, written for a ArcCatalog python addin. This is my first attempt to use Pro, without the convenience and the power we have with ArcCatalog (but hoping the idea: Add Stand Alone Data Catalog Like ArcCatalog to ArcGIS Pro will still happen so that this isn't as critical a test ....please vote up the idea! ).

I have several blahUtils.py type scripts to separate/organize my common functions. These are located in my Scripts folder in my addin file structure and I import into my tools/scripts using

from blahUtils import *
from moreBlahUtils import *
# etc...

rather than in a library in the python folder....that is, keep the addin all together.

This structure works well in ArcCatalog running is as an built/installed addin or from the Toolbox/Tool used to create the addin.

Putting aside the differences between Python 2.x and 3.x for the time being, how would I import the utility python scripts within my main tool script? I have read thru

Importing ArcPy—ArcPy Get Started | ArcGIS Desktop and

Python migration from 10.x to ArcGIS Pro—ArcPy Get Started | ArcGIS Desktop (no, I have not run the 2to3 tool yet)

but those seem to refer to fairly standard modules. I would like to keep my scripts in the folder structure for my toolbox (i.e. the addin structure) if possible.

The error I am getting

Traceback (most recent call last):
File "\\<server>\c$\Users\<user>\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\01-CopyDWCMasterSDEtoFGDB.py", line 47, in <module>
    from ADFGUtils import *
File "\\<server>\c$\Users\<user>\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py", line 134
    self.__dict__ = self"""
                         ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 188-189: truncated \UXXXXXXXX escape
Failed to execute (01-CopyDWCMasterSDEtoFGDB).

Is basically telling me it can not find the first custom script from ADFGutils import * I am trying to import.

Any suggestions on where I need to put these and how I would refer to them? Again, I prefer to keep in the same folder with/near my toolbox.

Thanks.

tagging Python‌ Python AddIns‌

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

DanPatterson_Retired — Sun, 12 Dec 2021 04:52:51 GMT

welcome to python 3... you are going to have to deal with the horrid "Users" folder so get used to it.

a = "c:\Users\Rebecca\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py"
  File "<ipython-input-50-c8a160e5d6d8>", line 1
    a = "c:\Users\Rebecca\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py"
       ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape‍‍‍‍‍

I took the liberty to make a 'fake' Rebecca path

What to do....

a = r"c:\Users\Rebecca\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py"
a
Out[54]: 'c:\\Users\\Rebecca\\_MyPyAddins\\dwcUpdateMasterFGDB\\Install\\scripts\\ADFGUtils.py'   # cool!!!

a = "c:\Users\Rebecca\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py".replace("\\","/")
  File "<ipython-input-52-86bfe186617b>", line 1
    a = "c:\Users\Rebecca\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py".replace("\\","/")
       ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape   # not so cool!!!‍‍‍‍‍‍‍‍‍

I don't know your work environment enough, but PRO uses python 3.5 currently and you will have to modify accordingly... got some blog link on the topic.

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

RebeccaStrauch__GISP — Wed, 19 Apr 2017 01:49:24 GMT

Thanks Dan, I have already been scanning thru your py blogs, so don't think I'm ignoring your hard work. I'll be testing more in the morning.

that's a shame that I hAve to put in the full path to import modes, but I've been building the paths for some oother needs (creating a temp .sde connection file and deleting after) so if putting in the path works, I can dal wil that I think. I was hoping of course for some way to reference it with relative paths. Do you know of anyway to do that with pro and Python 3.x?

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

DanPatterson_Retired — Sun, 12 Dec 2021 04:52:53 GMT

So it doesn't get lost... edit any config and/or import files accordingly... numpy etc etc uses this often ...

Think ... forward... not backwards... (kind of like pythonic thinking )

a = "c:/Users/Rebecca/_MyPyAddins/dwcUpdateMasterFGDB/Install/scripts/ADFGUtils.py"

a
Out[58]: 'c:/Users/Rebecca/_MyPyAddins/dwcUpdateMasterFGDB/Install/scripts/ADFGUtils.py'‍‍‍‍

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

RebeccaStrauch__GISP — Sun, 12 Dec 2021 04:52:56 GMT

Hi Dan, just testing in the Pro python window.

Have you tried importing a module with a full path? Is that possible?

fyi - I am using a UNC path, not a local folder, since this is not an addin and is being stored/editing on a server drive. So I can set my path, and verify the .py file is there, but when I try to import, it gives me a

ImportError: not module name '<myVariablename>'

How can I get the import command to read the full path? Is there a way to set another "mod location" BTW, I tried it as a local (non UNC path) and had the same error, and I used the arcpy.Exists() to verify that the python window could find the .py file.

scriptPath = r"\\dfgancgisdev3\c$\Users\rastrauch\_MyPyAddins\dwcUpdateMasterFGDB_Pro\Install\scripts"
print("the scriptPath is: {0}".format(scriptPath))
the scriptPath is: \\dfgancgisdev3\c$\Users\rastrauch\_MyPyAddins\dwcUpdateMasterFGDB_Pro\Install\scripts
moduleName = (os.path.join(scriptPath, "ADFGUtils")).replace("\\", "/")
print("the mod name is: {0}".format(moduleName))
the mod name is: //dfgancgisdev3/c$/Users/rastrauch/_MyPyAddins/dwcUpdateMasterFGDB_Pro/Install/scripts/ADFGUtils
from moduleName import *
Runtime error 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'moduleName'
from ("{0}".format(moduleName)) import *
Parsing error 
SyntaxError: invalid syntax (<string>, line 1)
moduleName2 = (os.path.join(scriptPath, "ADFGUtils.py")).replace("\\", "/")
from ("{0}".format(moduleName2)) import *
Parsing error 
SyntaxError: invalid syntax (<string>, line 1)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

(wow, Python syntax highlighter doesn't work well with Pro window output)

Any more suggestions?

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

RebeccaStrauch__GISP — Wed, 19 Apr 2017 17:37:21 GMT

I may have found something..checking it out now

python - How to import a module given the full path? - Stack Overflow

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

DanPatterson_Retired — Wed, 19 Apr 2017 17:56:09 GMT

Rebecca... Sorry, I have little to zilch experience having to work from network drives... The only thing I can do is forward you to the python documentation on the os.path module since os.join is an obvious backup to collecting paths. There have been changes between python 3.5 and python 3.6. you may want to flip back and forth between the versions. I suspect that is one reason that esri didn't port python 3.6 with pro 1.4.1. It will be interesting if any of the skulking insiders can shed some light on the current and future state.

But in short...

raw formatting when appropriate works IFFF you can hard code your paths
forward slashes are indestructible and work under all situations
if you have to use backslashes and omit a 'r' then you are going to get bitten by \a, \n, \t, \u, \U and a few others I have forgotten.... oh yeah... spaces
saying in our teaching lab... Users is for losers... in other words, if things are going to go wrong pathwise, something is pointing to a users folder
imports are guaranteed when the module/script you are importing is in the same script that is running
you need to set up an environment variable to point to a specific folder
for the pro anaconda distribution of python, there is still a site packages folder there... that folder is always looked at. for example... I installed PRO in an ArcPro folder.... here is what I got...

'C:\\ArcPro\\Resources\\ArcToolBox\\Scripts',
'C:\\ArcPro\\Resources\\ArcPy',
'C:\\Git_Dan', ############# added by me, 'stuff' from here always imports
'C:\\ArcPro\\bin\\Python\\envs\\arcgispro-py3\\python35.zip',
'C:\\ArcPro\\bin\\Python\\envs\\arcgispro-py3\\DLLs',
'C:\\ArcPro\\bin\\Python\\envs\\arcgispro-py3\\lib',
'C:\\ArcPro\\bin\\Python\\envs\\arcgispro-py3',
'C:\\ArcPro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages', ############ as does 'stuff' I installed here
'C:\\ArcPro\\bin',

Now how you tie your relative path thing into an absolute path structure is for those that have to work within a network structure, so you and others would know better. I install my own packages (and standalone scripts) in my GIT_Dan folder, I install other 'stuff' in the ... site-packages folder (ie pythonwin etc)

Maybe rscheitlin‌ might know or some of the other folks on the mvp/mod list.

Just be aware of the changes between 2.7 and 3.x and the changes between 3.5 and 3.6/7

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

RebeccaStrauch__GISP — Sun, 12 Dec 2021 04:52:59 GMT

Update: I put in a call to tech support to see if they can help. Good suggestion from Dan and python - How to import a module given the full path? - Stack Overflow post I listed above, however, I'm not getting it to work.

As a summary for some of what I have tested from that (and looking thru the help, docs, and other SE and other threads)

I was working with the first suggestion and one further down in the list re: adding to the system path. Neight worked, even if I did have my paths with the correct "/", etc.

import importlib.util
spec = importlib.util.spec_from_file_location("module.name", "/path/to/file.py")
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
foo.MyClass()‍‍‍‍‍

I got what looked to be close with this, but was getting a variety of errors (not worth listing here right now).

The suggestion of adding the path to the sys.path was more straight forward, but my implementation still gives me errors, mainly about unicode.

import sys
# the mock-0.3.1 dir contains testcase.py, testutils.py & mock.py
sys.path.append('/foo/bar/mock-0.3.1')

from testcase import TestCase
from testutils import RunTests
from mock import Mock, sentinel, patch‍‍‍‍‍‍‍

Even though my path has the "/", the import command appends "\MyScript.py" with the "\" to the module path. See below

import arcpy

def myMsgs(message):
     arcpy.AddMessage(message)
     print(message)

# Set the necessary product code
arcpy.SetProduct("ArcInfo")

import sys
pyVer = (sys.version[0:3])
myMsgs("the python version is: {0} ".format(pyVer))

if float(pyVer) == 3.5:
     myMsgs("Using Pro, python version {0}".format(pyVer))
     setVer = "3.5"
elif int(pyVer) == 2:
     myMsgs("Using Desktop, python version {0}".format(pyVer))
     setVer = "3.5"
else:
     myMsgs("Software unknown, but using python version {0}".format(pyVer))
     setVer = "other"

scriptPath = r"\\dfgancgisdev3\c$\Users\rastrauch\_MyPyAddins\dwcUpdateMasterFGDB_Pro\Install\scripts"
scriptPath35 = (scriptPath.replace("\\", "/"))

myMsgs(scriptPath35)
myMsgs("Do both paths exist?\n    scriptPath {0}\n  scriptPath35 {1}".format(arcpy.Exists(scriptPath), arcpy.Exists(scriptPath35)))
the python version is: 3.5 
Using Pro, python version 3.5
//dfgancgisdev3/c$/Users/rastrauch/_MyPyAddins/dwcUpdateMasterFGDB_Pro/Install/scripts
Do both paths exist?
    scriptPath True
  scriptPath35 True
sys.path.append(scriptPath35)
sys.path
['c:\\program files\\arcgis\\pro\\Resources\\arcpy', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\DLLs', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib', 'C:\\Program Files\\ArcGIS\\Pro\\bin', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages', 'C:\\Program Files\\ArcGIS\\Pro\\bin', 'C:\\Program Files\\ArcGIS\\Pro\\Resources\\ArcPy', 'C:\\Program Files\\ArcGIS\\Pro\\Resources\\ArcToolbox\\Scripts', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages\\win32', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages\\win32\\lib', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages\\Pythonwin', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages\\setuptools-27.2.0-py3.5.egg', 'c:\\program files\\arcgis\\pro\\Resources\\ArcToolbox\\Scripts', 'c:\\program files\\arcgis\\pro\\bin', '//dfgancgisdev3/c$/Users/rastrauch/_MyPyAddins/dwcUpdateMasterFGDB_Pro/Install/scripts']
from ADFGUtils import *
Runtime error 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "//dfgancgisdev3/c$/Users/rastrauch/_MyPyAddins/dwcUpdateMasterFGDB_Pro/Install/scripts\ADFGUtils.py", line 134
    self.__dict__ = self"""
                         ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 188-189: truncated \UXXXXXXXX escape
import ADFGUtils
Runtime error 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "//dfgancgisdev3/c$/Users/rastrauch/_MyPyAddins/dwcUpdateMasterFGDB_Pro/Install/scripts\ADFGUtils.py", line 134
    self.__dict__ = self"""
                         ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 188-189: truncated \UXXXXXXXX escape

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

What looks very promising in arcpy,

arcpy.env.scriptWorkspace

seems to take on the value of the arcpy.env.workspace but doesn't help with the loading of my mods. Not much additional info is in the Pro help re: the scriptWorkspace

Using environment settings in Python—Geoprocessing and Python | ArcGIS Desktop

Listing tools, toolboxes, and environment settings—Geoprocessing and Python | ArcGIS Desktop

I have a call in with tech support. Maybe this is something that isn't yet implemented.

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

DanPatterson_Retired — Wed, 19 Apr 2017 23:58:23 GMT

Addins? what about this from the Project section. I don't use addins so I can test and I don't know if you can add one of those non-local folders. Worth a look

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

RebeccaStrauch__GISP — Thu, 20 Apr 2017 01:00:14 GMT

I've looked at that, but it doesn't recognize the .addin file. I'm trying not to have to re-write or package everything. Tweaking to get the tools to work with python 3.x might take some work, but in theory, most of the tools should work. But we'll see.

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

curtvprice — Thu, 20 Apr 2017 01:42:45 GMT

Pro does not support Python addins. The only flavor it supports are those built in .NET.

Re: Import custom arcpy modules in Pro 1.4 (from Python addin structure)

RebeccaStrauch__GISP — Sun, 12 Dec 2021 04:53:02 GMT

For the purposes of closing this thread, below is one way to add a single custom mod which is stored in same folder as the script that is running. This is how my python addins are setup, so that is what I am testing. Posting here in case others are trying to do the same. I will start another post with this as a starting point, but hopefully to make it a bit more dynamic. (I will come back and add a link to that once posted.)

This is a cut and paste from a large working tool...if something doesn't make sense, let me know...

import os
import sys
import arcpy
# --->  This is for python 3.5 in Pro 1.4.1

# shows messages for python window or tool
def myMsgs(message):
     arcpy.AddMessage(message)
     print(message)
     
scriptPath, scriptName = (os.path.dirname(sys.argv[0]), os.path.basename(sys.argv[0]))
scriptPathforVer = (scriptPath.replace("\\", "/"))
arcpy.env.scriptWorkspace = scriptPathforVer
myMsgs("Path {0} exists? {1}\n".format(scriptPathforVer, arcpy.Exists(scriptPathforVer)))
# not sure whether this is still needed....
sys.path.append(scriptPathforVer)

mymod = 'myCustomMod'    # for a mod myCustomMod.py in same folder as this script

import importlib.util
modPath = os.path.join(scriptPathforVer, (mymod + ".py"))
modPath = (modPath.replace("\\", "/"))
myMsgs("modpath {0} exists? {1}".format(modPath, arcpy.Exists(modPath)))     
try:
     spec = importlib.util.spec_from_file_location(mymod, modPath)
     # hardcodes the myCustomMod mod name.  Refer to mod is myCustomMod.afunc()
     myCustomMod = importlib.util.module_from_spec(spec)
     spec.loader.exec_module(vars()['myCustomMod'])
     myMsgs("Successfully loaded {0}\n".format(mymod))
except:
     myMsgs("-> Unable to load {0}\n".format(mymod))     


# test it with something in the mod, e.g.
#  myCustomMod.afunc()
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

hope this is helpful to someone.