Using Enthought Canopy with ArcGIS/ArcPy

JordanWeaver · ‎04-22-2013

What is the best method for developing Python scripts for ArcGIS in a third-party installation of Python and IDE? I have tried using the suggested workflow from ESRI... that being the included installation of Python and PythonWin for the IDE. Unfortunately, these will not be sufficient or enjoyable so I was hoping to use the Canopy (formerly EPD) IDE (utilizing PyLab, etc.) for development with access to those elements that are available in the ArcGIS "preferred" environment.

I'm sure there are multiple steps to accomplishing this, and one problem may be that I have the 64-bit installation of Canopy (which I can uninstall and install the 32-bit version if necessary), but I was wondering if anyone has attempted to tackle this. I didn't want to start experimenting with environment variables until I was sure that I had the appropriate installations, as well as an understanding of what I was trying to accomplish.

Thanks in advance for any help with this problem.

Setup:
Windows 7
ArcGIS Desktop 10.1
Enthought Canopy (Python 2.7.3 64-bit)

I apologize if this has been answered previously, but I couldn't find an existing good explanation on the forum. Please redirect if solution already exists.

curtvprice · ‎04-22-2013

Here's my response to Jordan copied from my email earlier today, with some edits for clarity:

Jordan, I totally agree that it's tricky to integrate the two environments. Haven't tried Canopy yet, but I understand Canopy is pretty much the same with a juiced up UI and improved setup tools.

I really do believe that the best way to handle this is to install python from the ArcGIS distribution, and separately install Canopy. (You need EPD 32-bit to import 32-bit arcpy and EPD 64-bit to import 64-bit arcpy.) If you install EPD 32 and 64 and ArcGIS Desktop x64 background geoprocessing, this adds up to no less than four python distributions to deal with.

Jason Pardy from Esri suggested I just try putting the site-packages folders in the python path. From ArcGIS I have been successful accessing EPD packages by adding a similar .pth file to the ArcGIS python install site-packages folder. I haven't run into any problems with "circular searches", I think Python is smart enough not to look in a site-packages folder twice.

One way to modify the Python path is to copy the file Desktop.pth from \Python27\ArcGIS10.1\lib\site-packages to the file zzArcGIS.pth in the 32-bit Canopy site-packages folder (the paths are loaded in alpha order). This should allow you to import arcpy just fine on the Canopy side. Another approach is to just add the paths to the PYTHONPATH variable (see bat scripts below).

bat scripts are especially useful if you find you need full control of the PATH and other environment variables to get things to work right. I've attached an example. The way it's done in UNIX -- much less ugly than registering DLLs and messing with globally-applied environment variables! (When you run the scripts, the environment tweaks only apply to the shell and its children, not your global environment.) A few examples are attached:

:: epd32arc.bat
:: start EPD32 python prompt with arcpy
@echo off
set EPDPATH=E:\python27_epd32
set AGSPATH=D:\ArcGIS\Desktop10.1
:: save paths
set PATHENV=%PATH%
set PPATHENV=%PYTHONPATH%
:: set paths EPD with access to arcpy
set PATH=%EPDPATH%;%EPDPATH%\scripts;%PATH%
set PYTHONPATH=%PYTHONPATH%;%AGSPATH%\bin;%AGSPATH%\arcpy;%AGSPATH%\ArcToolbox\Scripts
cmd /c %EPDPATH%\python.exe
:: restore paths
set PATH=%PATHENV%
set PYTHONPATH=%PPATHENV%
:: delete variables
set EPDPATH=
set AGSPATH=
set PATHENV=
set PPATHENV=

:: arcmap_epd.bat - start ArcMap with EPD32 libraries available
@echo off
:: set the paths below to your install locations
set EPDPATH=E:\python27_epd32
set AGSPATH=D:\ArcGIS\Desktop10.1
:: save paths
set PATHENV=%PATH%
set PPATHENV=%PYTHONPATH%
:: set paths for ArcMap w/ EPD
set PATH=%EPDPATH%;%EPDPATH%\scripts;%PATH%
set PYTHONPATH=%PYTHONPATH%;%EPDPATH%\lib\site-packages
echo Starting ArcMap w/ Enthought Python Distribution modules...
start /b /d %AGSPATH%\bin ArcMap.exe
:: restore paths
set PATH=%PATHENV%
set PYTHONPATH=%PPATHENV%
:: delete variables
set EPDPATH=
set AGSPATH=
set PATHENV=
set PPATHENV=

JordanWeaver · ‎04-23-2013

I ended up uninstalling all Enthought builds I had on my system and started from scratch with a single 32-bit Canopy installation. From there, I copied the Desktop10.1.pth file from C:\Python27\ArcGIS10.1\Lib\site-packages to the Canopy site-packages directory, renaming it zzArcGIS.pth as you suggested (for my system it was located: C:\Users\<user>\AppData\Local\Enthought\Canopy32\System\Lib\site-packages).

This worked great and I�??m surprised that it ended up being this easy. I�??m sort of out of my element�?� I traditionally work in UNIX/OS X so I appreciate the included bat scripts. Since the other .pth solution seems to get me everything I was looking for (i.e. accessing/using ArcPy from within Canopy), I will avoid messing with the Windows environment variables as I�??ve had that go sour for other things despite my best efforts to manage them correctly.

Thanks again for your help and I'll post any updates on unexpected issues/behavior.

curtvprice · ‎04-25-2013

Here's a better description of using pth files from the gis.stackexchange.com site, posted by my colleague Rich Signell. If you don't want to tweak your environment for your python session, this is an easier approach than a .bat file. Note path files are read in alpha order, so if you name your custom pth file starting with "zz" it will be loaded last -- which may avoid conflicts.

[/hr]

The Enthought Python Distributions, including Canopy, are stand-alone and don't need anything in the registry or in the environment variables. So they can play nice with other Python distributions like the one that comes with ArcGIS 10.1.

To access ArcGIS 10.1 python modules from Enthought Canopy: Create a "path" file in the Canopy user directory that points to the ArcGIS 10.1 python modules. On my system, I created a text file called arcpy.pth with these 4 lines:

# .pth file for arcpy
C:\ArcGIS\Desktop10.1\bin
C:\ArcGIS\Desktop10.1\arcpy
C:\ArcGIS\Desktop10.1\ArcToolbox\Scripts

and put arcpy.pth in the Canopy User site-packages directory:

C:\Users\rsignell\AppData\Local\Enthought\Canopy32\User\lib\site-packages

To access Enthought Canopy modules from ArcGIS 10.1: Create a "path" file in the Arc site-packages folder that points to the Canopy python modules. On my system, I created a text file called epd.pth with these 3 lines:

# .pth file for EPD Canopy
C:\Users\rsignell\AppData\Local\Enthought\Canopy32\User\Lib\site-packages
C:\Users\rsignell\AppData\Local\Enthought\Canopy32\System\Lib\site-packages

and then put epd.pth in the Arc site-package folder:

C:\Python27\ArcGIS10.1\Lib\site-packages

This works because Enthought Canopy and ArcGIS10.1 are at the same version of Python (2.7) and Numpy (1.6). I hope we can continue to have this compatibility!

NielsClaes · ‎12-15-2013

I changed to the 32 version of Canopy and copied the pathfile into the 'site packages' folder as suggested.
When importing now arcpy, I get the following error:

'import arcpy
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-50d283eebbb3> in <module>()
----> 1 import arcpy

C:\Program Files (x86)\ArcGIS\Desktop10.2\arcpy\arcpy\__init__.py in <module>()
     19 import warnings
     20
---> 21 from arcpy.geoprocessing import gp
     22 from arcpy.geoprocessing import env
     23 from arcpy.geoprocessing._base import gptooldoc as _gptooldoc
...
'
ending with a Runtime Error: Not Initialized

Does anyone had the same problem or an idea how to solve it?

Best Regards

curtvprice · ‎12-16-2013

As noted above, we've gotten a little fancier with this. The .pth files in the site-packages folder are loaded in alphabetical order so we've been naming these files "zzarcpy.pth" (in the canopy site-packages) and "zzepd.pth" (in the ArcGIS site-packages). Naming them with a "zz" prefix should ensure they are listed after the "native" modules for that python install. (First folder in the list wins, just like the Windows and Unix PATH variables.)

Also double check to make sure you don't have any settings in your windows environment PYTHONPATH.

If you're still stuck, have your test script dump out the sys.path before you import arcpy, that may help see what's going wrong with your sys.path. Hopefully it will become clear.

import sys
k = 0
for p in sys.path:
  print "{0:4} {1}".format(k,p)
  k += 1

Here's an example - on this particular machine I don't have Enthought Canopy installed. But if it was, I'd want to see those paths AFTER the ArcGIS ones to avoid conflicts when importing arcpy.

Remember if you do run into conflicts, there is a workaround - you can modify the sys.path if, for example, you want to temporarily knock out the ArcGIS references to use some Canopy modules that have conflicts with the python versions Esri is using.

>>> k = 0
>>> for p in sys.path:
...   print "{0:4} {1}".format(k,p)
...   k += 1
...
   0 (os.curdir - current python working folder)
   1 C:\ArcGIS\Desktop10.1\ArcToolbox\Toolboxes\USGS_EGISTools\scripts
   2 C:\Windows\system32\python27.zip
   3 C:\Python27\ArcGIS10.1\DLLs
   4 C:\Python27\ArcGIS10.1\lib
   5 C:\Python27\ArcGIS10.1\lib\plat-win
   6 C:\Python27\ArcGIS10.1\lib\lib-tk
   7 C:\Python27\ArcGIS10.1
   8 C:\Python27\ArcGIS10.1\lib\site-packages
   9 C:\ArcGIS\Desktop10.1\bin
  10 C:\ArcGIS\Desktop10.1\arcpy
  11 C:\ArcGIS\Desktop10.1\ArcToolbox\Scripts
>>> import arcpy
>>> arcpy.__file__
'C:\\ArcGIS\\Desktop10.1\\arcpy\\arcpy\\__init__.py'

ScottStephenson · ‎11-23-2015

Curtis, thanks for your help with this. I'm still experiencing the same error as others in this thread (ImportError: DLL load failed: %1 is not a valid Win32 application) after adding the arcpy.pth files to the Canopy site packages directory as instructed above. I dumped out the sys.path (copied below), and as you can see, there are Canopy paths both before and after the ArcGIS paths. Do you know how to get the Canopy paths to load after the ArcGIS paths, as you suggested?

0 C:\Users\scott.stephenson\Dropbox\Python\scripts\Emissions
      1
      2 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\User\Scripts\python27.zip
      3 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.5.3123.win-x86_64\DLLs
      4 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.5.3123.win-x86_64\lib
      5 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.5.3123.win-x86_64\lib\plat-win
      6 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.5.3123.win-x86_64\lib\lib-tk
      7 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.5.3123.win-x86_64
      8 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\User
      9 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\User\lib\site-packages
    10 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\User\lib\site-packages\PIL
    11 C:\Program Files (x86)\ArcGIS\Desktop10.2\bin
    12 C:\Program Files (x86)\ArcGIS\Desktop10.2\arcpy
    13 C:\Program Files (x86)\ArcGIS\Desktop10.2\ArcToolbox\Scripts
    14 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\User\lib\site-packages\win32
    15 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\User\lib\site-packages\win32\lib
    16 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\User\lib\site-packages\Pythonwin
    17 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata
    18 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.5.3123.win-x86_64\lib\site-packages
    19 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.5.3123.win-x86_64\lib\site-packages\win32
    20 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.5.3123.win-x86_64\lib\site-packages\win32\lib
    21 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.5.3123.win-x86_64\lib\site-packages\Pythonwin
    22 C:\Users\scott.stephenson\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.5.3123.win-x86_64\lib\site-packages\IPython\extensions
    23 C:\Users\scott.stephenson\.ipython

curtvprice · ‎11-25-2015

Did you:

1) make sure the canopy path is not in the system PATH (this is included in the python sys.path)

2) name the canopy .pth file in /lib/site-packages starting with zz so it is loaded last.

I have moved on to using anaconda (free) but this approach is still working for me. I use an environment set up specifically compatible with ArcGIS ("arc1022") and then I can also have other environments to work with non-ArcGIS compatible packages.

ScottStephenson · ‎11-27-2015

Thanks again Curtis. I don't know how to check if the canopy path is not in the system PATH. I named the .pth files (both zzarcpy.pth in the Canopy site-packages folder, and zzcanopy.pth in the Python27 site-packages folder) accordingly so they would be loaded last. No luck

I've also tried installing and running the 32-bit version of Canopy -- no luck there either.

Would you recommend I try using Anaconda? If it's easier to get arcpy to load in Anaconda, I will gladly give that a try.

DanPatterson_Retired · ‎11-27-2015

Go with what works without problems or special considerations... people can weigh in if they have information to the contrary

PythonWin
PyScripter

I have both versions installed for: (associated modules in brackets)

Python 2.7.x ... aka ArcMap up to 10.3.1 (Numpy, MatPlotLib, appropriate python versioned)
Python 3.4.x ... aka ArcGIS Pro to version 1.1.1 (Numpy,SciPy, SymPy, MatPlotLib,Pandas...etc blah blah)

Amazingly, if you let Arcmap and ArcGIS Pro install the version of python etc FIRST, PythonWin and PyScripter find them and their appropriate version without any floor dancing about paths etc.

So unless you have a mission critical reason to use something else, you might use the current known-to-work IDE's and wait until IronPython comes along (or so is rumoured)

Back to you Curtis for counter-point