Import custom arcpy modules in Pro 1.4 (from Python addin structure)

[RebeccaStrauch__GISP]

I'm trying to test some of my custom tools, written for a ArcCatalog python addin.   This is my first attempt to use Pro, without the convenience and the power we have with ArcCatalog (but hoping the idea: Add Stand Alone Data Catalog Like ArcCatalog to ArcGIS Pro  will still happen so that this isn't as critical a test ....please vote up the idea! ).

I have several blahUtils.py  type scripts to separate/organize my common functions. These are located in my Scripts folder in my addin file structure and I import into my tools/scripts using

from blahUtils import *

from moreBlahUtils import *

# etc...

rather than in a library in the python folder....that is, keep the addin all together.

This structure works well in ArcCatalog running is as an built/installed addin or from the Toolbox/Tool used to create the addin.

Putting aside the differences between Python 2.x and 3.x for the time being, how would I import the utility python scripts within my main tool script?  I have read thru

Importing ArcPy—ArcPy Get Started | ArcGIS Desktop    and

Python migration from 10.x to ArcGIS Pro—ArcPy Get Started | ArcGIS Desktop   (no, I have not run the 2to3 tool yet)

but those seem to refer to fairly standard modules.  I would like to keep my scripts in the folder structure for my toolbox (i.e. the addin structure) if possible.

The error I am getting

Traceback (most recent call last):
  File "\\<server>\c$\Users\<user>\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\01-CopyDWCMasterSDEtoFGDB.py", line 47, in <module>
    from ADFGUtils import *
  File "\\<server>\c$\Users\<user>\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py", line 134
    self.__dict__ = self"""
                         ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 188-189: truncated \UXXXXXXXX escape
Failed to execute (01-CopyDWCMasterSDEtoFGDB).

Is basically telling me it can not find the first custom script from ADFGutils import * I am trying to import. 

Any suggestions on where I need to put these and how I would refer to them?  Again, I prefer to keep in the same folder with/near my toolbox.

Thanks.

tagging Python‌ Python AddIns‌

[DanPatterson_Retired]

welcome to python 3... you are going to have to deal with the horrid "Users" folder so get used to it.

a = "c:\Users\Rebecca\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py"
  File "<ipython-input-50-c8a160e5d6d8>", line 1
    a = "c:\Users\Rebecca\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py"
       ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape‍‍‍‍‍

I took the liberty to make a 'fake' Rebecca path

What to do....

a = r"c:\Users\Rebecca\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py"
a
Out[54]: 'c:\\Users\\Rebecca\\_MyPyAddins\\dwcUpdateMasterFGDB\\Install\\scripts\\ADFGUtils.py'   # cool!!!

a = "c:\Users\Rebecca\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py".replace("\\","/")
  File "<ipython-input-52-86bfe186617b>", line 1
    a = "c:\Users\Rebecca\_MyPyAddins\dwcUpdateMasterFGDB\Install\scripts\ADFGUtils.py".replace("\\","/")
       ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape   # not so cool!!!‍‍‍‍‍‍‍‍‍

I don't know your work environment enough, but PRO uses python 3.5 currently and you will have to modify accordingly... got some blog link on the topic.

[RebeccaStrauch__GISP]

Thanks Dan, I have already been scanning thru your py blogs, so don't think I'm ignoring your hard work. I'll be testing more in the morning.

that's a shame that I hAve to put in the full path to import modes, but I've been building the paths for some oother needs (creating a temp .sde connection file and deleting after) so if putting in the path works, I can dal wil that I think.  I was hoping of course for some way to reference it with relative paths.  Do you know of anyway to do that with pro and Python 3.x?

[DanPatterson_Retired]

So it doesn't get lost... edit any config and/or import files accordingly... numpy etc etc uses this often ...

Think ... forward... not backwards... (kind of like pythonic thinking )

a = "c:/Users/Rebecca/_MyPyAddins/dwcUpdateMasterFGDB/Install/scripts/ADFGUtils.py"

a
Out[58]: 'c:/Users/Rebecca/_MyPyAddins/dwcUpdateMasterFGDB/Install/scripts/ADFGUtils.py'‍‍‍‍

[RebeccaStrauch__GISP]

Hi Dan, just testing in the Pro python window. 

Have you tried importing a module with a full path?  Is that possible?

fyi - I am using a UNC path, not a local folder, since this is not an addin and is being stored/editing on a server drive.  So I can set my path, and verify the .py file is there, but when I try to import, it gives me a

ImportError: not module name '<myVariablename>'

How can I get the import command to read the full path? Is there a way to set another "mod location"    BTW, I tried it as a local (non UNC path) and had the same error, and I used the arcpy.Exists() to verify that the python window could find the .py file.

scriptPath = r"\\dfgancgisdev3\c$\Users\rastrauch\_MyPyAddins\dwcUpdateMasterFGDB_Pro\Install\scripts"
print("the scriptPath is: {0}".format(scriptPath))
the scriptPath is: \\dfgancgisdev3\c$\Users\rastrauch\_MyPyAddins\dwcUpdateMasterFGDB_Pro\Install\scripts
moduleName = (os.path.join(scriptPath, "ADFGUtils")).replace("\\", "/")
print("the mod name is: {0}".format(moduleName))
the mod name is: //dfgancgisdev3/c$/Users/rastrauch/_MyPyAddins/dwcUpdateMasterFGDB_Pro/Install/scripts/ADFGUtils
from moduleName import *
Runtime error 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'moduleName'
from ("{0}".format(moduleName)) import *
Parsing error 
SyntaxError: invalid syntax (<string>, line 1)
moduleName2 = (os.path.join(scriptPath, "ADFGUtils.py")).replace("\\", "/")
from ("{0}".format(moduleName2)) import *
Parsing error 
SyntaxError: invalid syntax (<string>, line 1)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

(wow, Python syntax highlighter doesn't work well with Pro window output)

Any more suggestions?

[RebeccaStrauch__GISP]

I may have found something..checking it out now

python - How to import a module given the full path? - Stack Overflow 

[DanPatterson_Retired]

Rebecca... Sorry, I have little to zilch experience having to work from network drives...  The only thing I can do is forward you to the python documentation on the os.path module since os.join is an obvious backup to collecting paths.  There have been changes between python 3.5 and python 3.6. you may want to flip back and forth between the versions.  I suspect that is one reason that esri didn't port python 3.6 with pro 1.4.1.  It will be interesting if any of the skulking insiders can shed some light on the current and future state.

But in short...

• raw formatting when appropriate works IFFF you can hard code your paths
• forward slashes are indestructible and work under all situations
• if you have to use backslashes and omit a 'r' then you are going to get bitten by \a, \n, \t, \u, \U and a few others I have forgotten.... oh yeah... spaces
• saying in our teaching lab... Users is for losers... in other words, if things are going to go wrong pathwise, something is pointing to a users folder
• imports are guaranteed when the module/script you are importing is in the same script that is running
• you need to set up an environment variable to point to a specific folder
• for the pro anaconda distribution of python, there is still a site packages folder there... that folder is always looked at.   for example... I installed PRO in an ArcPro folder.... here is what I got...

 'C:\\ArcPro\\Resources\\ArcToolBox\\Scripts',
 'C:\\ArcPro\\Resources\\ArcPy',
 'C:\\Git_Dan',                ############# added by me, 'stuff' from here always imports
 'C:\\ArcPro\\bin\\Python\\envs\\arcgispro-py3\\python35.zip',
 'C:\\ArcPro\\bin\\Python\\envs\\arcgispro-py3\\DLLs',
 'C:\\ArcPro\\bin\\Python\\envs\\arcgispro-py3\\lib',
 'C:\\ArcPro\\bin\\Python\\envs\\arcgispro-py3',
 'C:\\ArcPro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages',  ############ as does 'stuff' I installed here
  'C:\\ArcPro\\bin',

Now how you tie your relative path thing into an absolute path structure is for those that have to work within a network structure, so you and others would know better.  I install my own packages (and standalone scripts) in my GIT_Dan folder, I install other 'stuff' in the ... site-packages folder (ie pythonwin etc)

Maybe rscheitlin‌ might know or some of the other folks on the mvp/mod list.

Just be aware of the changes between 2.7 and 3.x and the changes between 3.5 and 3.6/7

[RebeccaStrauch__GISP]

Update: I put in a call to tech support to see if they can help.  Good suggestion from Dan and  python - How to import a module given the full path? - Stack Overflow  post I listed above, however, I'm not getting it to work.

As a summary for some of what I have tested from that (and looking thru the help, docs, and other SE and other threads)

I was working with the first suggestion and one further down in the list re: adding to the system path.  Neight worked, even if I did have my paths with the correct "/", etc.

import importlib.util
spec = importlib.util.spec_from_file_location("module.name", "/path/to/file.py")
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
foo.MyClass()‍‍‍‍‍

I got what looked to be close with this, but was getting a variety of errors (not worth listing here right now).

The suggestion of adding the path to the sys.path was more straight forward, but my implementation still gives me errors, mainly about unicode.

import sys
# the mock-0.3.1 dir contains testcase.py, testutils.py & mock.py
sys.path.append('/foo/bar/mock-0.3.1')

from testcase import TestCase
from testutils import RunTests
from mock import Mock, sentinel, patch‍‍‍‍‍‍‍

Even though my path has the "/", the import command appends "\MyScript.py" with the "\" to the module path.  See below

import arcpy

def myMsgs(message):
     arcpy.AddMessage(message)
     print(message)

# Set the necessary product code
arcpy.SetProduct("ArcInfo")

import sys
pyVer = (sys.version[0:3])
myMsgs("the python version is: {0} ".format(pyVer))

if float(pyVer) == 3.5:
     myMsgs("Using Pro, python version {0}".format(pyVer))
     setVer = "3.5"
elif int(pyVer) == 2:
     myMsgs("Using Desktop, python version {0}".format(pyVer))
     setVer = "3.5"
else:
     myMsgs("Software unknown, but using python version {0}".format(pyVer))
     setVer = "other"

scriptPath = r"\\dfgancgisdev3\c$\Users\rastrauch\_MyPyAddins\dwcUpdateMasterFGDB_Pro\Install\scripts"
scriptPath35 = (scriptPath.replace("\\", "/"))

myMsgs(scriptPath35)
myMsgs("Do both paths exist?\n    scriptPath {0}\n  scriptPath35 {1}".format(arcpy.Exists(scriptPath), arcpy.Exists(scriptPath35)))
the python version is: 3.5 
Using Pro, python version 3.5
//dfgancgisdev3/c$/Users/rastrauch/_MyPyAddins/dwcUpdateMasterFGDB_Pro/Install/scripts
Do both paths exist?
    scriptPath True
  scriptPath35 True
sys.path.append(scriptPath35)
sys.path
['c:\\program files\\arcgis\\pro\\Resources\\arcpy', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\DLLs', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib', 'C:\\Program Files\\ArcGIS\\Pro\\bin', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages', 'C:\\Program Files\\ArcGIS\\Pro\\bin', 'C:\\Program Files\\ArcGIS\\Pro\\Resources\\ArcPy', 'C:\\Program Files\\ArcGIS\\Pro\\Resources\\ArcToolbox\\Scripts', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages\\win32', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages\\win32\\lib', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages\\Pythonwin', 'C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\lib\\site-packages\\setuptools-27.2.0-py3.5.egg', 'c:\\program files\\arcgis\\pro\\Resources\\ArcToolbox\\Scripts', 'c:\\program files\\arcgis\\pro\\bin', '//dfgancgisdev3/c$/Users/rastrauch/_MyPyAddins/dwcUpdateMasterFGDB_Pro/Install/scripts']
from ADFGUtils import *
Runtime error 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "//dfgancgisdev3/c$/Users/rastrauch/_MyPyAddins/dwcUpdateMasterFGDB_Pro/Install/scripts\ADFGUtils.py", line 134
    self.__dict__ = self"""
                         ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 188-189: truncated \UXXXXXXXX escape
import ADFGUtils
Runtime error 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "//dfgancgisdev3/c$/Users/rastrauch/_MyPyAddins/dwcUpdateMasterFGDB_Pro/Install/scripts\ADFGUtils.py", line 134
    self.__dict__ = self"""
                         ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 188-189: truncated \UXXXXXXXX escape

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

What looks very promising in arcpy,

arcpy.env.scriptWorkspace

seems to take on the value of the arcpy.env.workspace   but doesn't help with the loading of my mods.  Not much additional info is in the Pro help re: the scriptWorkspace   

Using environment settings in Python—Geoprocessing and Python | ArcGIS Desktop 

Listing tools, toolboxes, and environment settings—Geoprocessing and Python | ArcGIS Desktop 

I have a call in with tech support. Maybe this is something that isn't yet implemented.

[DanPatterson_Retired]

Addins?  what about this from the Project section.  I don't use addins so I can test and I don't know if you can add one of those non-local folders.  Worth a look

[RebeccaStrauch__GISP]

I've looked at that, but it doesn't recognize the .addin file.   I'm trying not to have to re-write or package everything.  Tweaking to get the tools to work with python 3.x might take some work, but in theory, most of the tools should work.  But we'll see.