sys.path in ArcGIS python implementation (standalone - and in ArcGIS Desktop prompt)

[curtvprice]

ArcGIS 10.1 Final
Separate Python distribution (EPD) installed in its own folder (not in system PATH).

Here's what I see from a command line:

Z:\>d:\python27\ArcGIS10.1\python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print '\n'.join(sys.path)
  ("" - current folder)
D:\ArcGIS\Desktop10.1\ArcToolbox\Toolboxes\USGS_EGISTools\scripts # in my PYTHONPATH
C:\WINDOWS\system32\python27.zip
d:\python27\ArcGIS10.1\DLLs
d:\python27\ArcGIS10.1\lib
d:\python27\ArcGIS10.1\lib\plat-win
d:\python27\ArcGIS10.1\lib\lib-tk
d:\python27\ArcGIS10.1
d:\python27\ArcGIS10.1\lib\site-packages
D:\ArcGIS\Desktop10.1\bin       # Desktop10.1.pth in site-packages 
D:\ArcGIS\Desktop10.1\arcpy   # Desktop10.1.pth in site-packages
D:\ArcGIS\Desktop10.1\ArcToolbox\Scripts # Desktop10.1.pth in site-packages
>>>

But from within the ArcMap Python prompt I'm seeing paths that are not in my PYTHONPATH, somehow it's picking up libraries from my EPD.

>>> print '\n'.join(sys.path)
D:\ArcGIS\Desktop10.1\ArcToolbox\Toolboxes\USGS_EGISTools\scripts # in my PYTHONPATH
d:\arcgis\desktop10.1\arcpy
C:\WINDOWS\system32\python27.zip
E:\python27_epd32\Lib   # where did these three come from??
E:\python27_epd32\DLLs
E:\python27_epd32\Lib\lib-tk
D:\Python27\ArcGIS10.1\Lib
D:\Python27\ArcGIS10.1\DLLs
D:\Python27\ArcGIS10.1\Lib\lib-tk
C:\Documents and Settings\cprice # where did this come from? 
D:\ArcGIS\Desktop10.1\bin # Desktop10.1.pth in site-packages (why  here?)
D:\Python27\ArcGIS10.1
D:\Python27\ArcGIS10.1\lib\site-packages
D:\ArcGIS\Desktop10.1\arcpy # Desktop10.1.pth in site-packages
D:\ArcGIS\Desktop10.1\ArcToolbox\Scripts # Desktop10.1.pth in site-packages

My EPD install gives me a shorter list, but it's still mysterious to me:

Z:\>e:\python27_epd32\python.exe
Enthought Python Distribution -- www.enthought.com
Version: 7.3-2 (32-bit)

Python 2.7.3 |EPD 7.3-2 (32-bit)| (default, Apr 12 2012, 14:30:37) [MSC v
Type "credits", "demo" or "enthought" for more information.
>>> import sys
>>> print "\n".join(sys.path)
  ("" - current folder)
D:\ArcGIS\Desktop10.1\ArcToolbox\Toolboxes\USGS_EGISTools\scripts
e:\python27_epd32\python27.zip
e:\python27_epd32\DLLs
e:\python27_epd32\lib
e:\python27_epd32\lib\plat-win
e:\python27_epd32\lib\lib-tk
e:\python27_epd32
e:\python27_epd32\lib\site-packages
e:\python27_epd32\lib\site-packages\PIL   # PIL.pth in site-packages
e:\python27_epd32\lib\site-packages\win32 # pywin32.pth in site-packages
e:\python27_epd32\lib\site-packages\win32\lib # pywin32.pth in site-packages
e:\python27_epd32\lib\site-packages\Pythonwin # pywin32.pth in site-packages 

This leaves me with some question about the ArcGIS Desktop Python prompt.

1. There are some entries in the sys.path in both setups that I honestly don't see where they came from. The first one in the lists above came from my PYTHONPATH, and the the last three are in the .pth file in site-packages. Where did all the others entries come from?
2. There are some imports (sys, arcpy) that the ArcMap python window has loaded already. Where did that happen?
3. The ArcMap python prompt is picking up some paths from somewhere mysterious (in italic above)
4. In all setups, what's up with the path: C:\WINDOWS\system32\python27.zip. That file does not exist in my install.

[JasonScheirer · Accepted solution]

I'm more that a little curious about the motivations for needing to know this and dubious that it matters, but I'll indulge you:

1. There are some entries in the sys.path in both setups that I honestly don't see where they came from. The first one in the lists above came from my PYTHONPATH, and the the last three are in the .pth file in site-packages. Where did all the others entries come from?

From the bootstrapping process Python does when it first initializes itself in the process. There are other places Python looks (%PATH%, %PYTHONPATH%, registry keys, site.py and sitecustomize.py). Additional bin directories are injected in the path in the CPython code based on the current path.

2. There are some imports (sys, arcpy) that the ArcMap python window has loaded already. Where did that happen?

When the Geoprocessing framework bootstraps itself to run Python scripts. It imports some commonly-used modules for script tools/Field Calculator that users tend to forget to import just as a courtesy.

3. The ArcMap python prompt is picking up some paths from somewhere mysterious (in italic above)

The same items would be found in python.exe if it didn't know to short-circuit that process. Since the working directory of python.exe is c:\python27\arcgis10.1, it knows to ignore the bin dircetories of other Python 2.7 installs it finds. When it's running embedded in another program such as arcmap.exe, it isn't necessarily as sure as to which install is canonical so it errs on the side of inclusiveness. Again, look in the registry and environments in addition to where you've looked already

4. In all setups, what's up with the path: C:\WINDOWS\system32\python27.zip. That file does not exist in my install.

Legacy reasons. It's part of Python itself. I'm sure it's deprecated by now, I think as the current bootstrap process was being refined it was a part of it. Basically you can place a python27.zip in that location and fill it with .py files and they will show up as importable modules. For instance, you can place a site.py or sitecustomize.py in there that sets the sys.path and you'd get a uniform sys.path across the board in all places.

(cp - reformatted)

[curtvprice]

I'm more that a little curious about the motivations for needing to know this and dubious that it matters, but I'll indulge you.

Thank you Jason, I really appreciate this - it really gives me a better handle on what's happening here.

This is important to me because a colleague was having issues with the ArcMap prompt sys.path picking up a few 64 bit paths from his EPD 64 install - which could obviously break things (or not?). Knowing a bit more what is happening under the hood will definitely help us (and others trying to support multiple Pythons on the same computer) avoid problems.

In general we seem to be having the best luck when we use startup scripts to keep total control of the system PATH when starting a specific Python that we want.

I found some good documentation in the python manuals (27.14. site �?? Site-specific configuration hook), which taught me this cool trick to quickly check your setup:

D:\Python27\ArcGIS10.1>python -m site
sys.path = [
    'D:\\Python27\\ArcGIS10.1',
    'D:\\ArcGIS\\Desktop10.1\\ArcToolbox\\Toolboxes\\USGS_EGISTools\\scripts',
    'C:\\WINDOWS\\system32\\python27.zip',
    'D:\\Python27\\ArcGIS10.1\\DLLs',
    'D:\\Python27\\ArcGIS10.1\\lib',
    'D:\\Python27\\ArcGIS10.1\\lib\\plat-win',
    'D:\\Python27\\ArcGIS10.1\\lib\\lib-tk',
    'D:\\Python27\\ArcGIS10.1\\lib\\site-packages',
    'D:\\ArcGIS\\Desktop10.1\\bin',
    'D:\\ArcGIS\\Desktop10.1\\arcpy',
    'D:\\ArcGIS\\Desktop10.1\\ArcToolbox\\Scripts',
]
USER_BASE: 'C:\\Documents and Settings\\cprice\\Application Data\\Python' (doesn't exist)
USER_SITE: 'C:\\Documents and Settings\\cprice\\Application Data\\Python\\Python27\\site-packages' (doesn't exist)
ENABLE_USER_SITE: True

[JasonScheirer]

It's always a bit of a confusing hassle to manage multiple installations of the same software on one machine, I'm constantly running into this myself on 64 bit systems where I have 64 and 32 bit installs of the same thing, or the fact that TortoiseHg uses its own local Python, etc.

For further reference/esoterica, the two source files that drive the sys.path in Python 2.7.2:

http://hg.python.org/cpython/file/8527427914a2/PC/getpathp.c
http://hg.python.org/cpython/file/8527427914a2/Python/pythonrun.c

Follow the PySys_GetPath() calls to get some idea of what's going on.

[curtvprice]

It's always a bit of a confusing hassle to manage multiple installations of the same software on one machine, I'm constantly running into this myself on 64 bit systems where I have 64 and 32 bit installs of the same thing, or the fact that TortoiseHg uses its own local Python, etc.

I did find a spot in the registry that provides those standard extra sys.path entries (I'm on XP64, so it's in the Wow6432Node folder):

Name: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\2.7\PythonPath
Data: D:\Python27\ArcGIS10.1\Lib;D:\Python27\ArcGIS10.1\DLLs;D:\Python27\ArcGIS10.1\Lib\lib-tk