Problem importing arcpy in ArcMap console -- 64 bit DLL conflict

[JohnBurkhart1]

Hello,

Working on Windows 7 with ArcGIS 10.1 using the 64 bit geoprocessing python... I have been having problems with using python **inside** ArcMap. Outside of ArcMap, I can run the ArcGISx6410.1/python.exe without problems. Here is my WORKING case:

arcpython.bat which will launch 64 bit python and use 64 bit libraries/modules and wraps everything in Ipython.

@echo off set path=C:\Python27\ArcGISx6410.1;%path% set PYTHONPATH=C:\Python27\ArcGISx6410.1\Lib\site-packages\;C:\Python27\Lib\;C:\Python27\Lib\site-packages\;%pythonpath% C:\Python27\ArcGISx6410.1\python.exe -c "import sys; from IPython.frontend.terminal.ipapp import launch_new_instance; sys.exit(launch_new_instance())" %* exit /B %ERRORLEVEL%

With this, I get an Ipython window and all arcpy functionality with no errors.

However, when I launch the python console inside of ArcMap I get a traceback with the following ultimate error:

  File "C:\Python27\Lib\site-packages\numpy\lib\type_check.py", line 8, in <module>     import numpy.core.numeric as _nx   File "C:\Python27\Lib\site-packages\numpy\core\__init__.py", line 5, in <module>     import multiarray ImportError: DLL load failed: %1 is not a valid Win32 application. >>> 

So clearly, it's a problem with the 64 bit imports.

I have my PYTHONPATH set to point to my 64 bit libraries, and I've defined a pythonExePath to point to the ArcGISx6410.1/python.exe.

I'm stumped, because I don't understand how to control which python executable is called from **WITHIN** Arcmap. Output from the console only provides this:

>>> import sys >>> sys.executable  'C:\\Program Files (x86)\\ArcGIS\\Desktop10.1\\bin\\ArcMap.exe' >>> 

Please advise on how to instruct the console to use the ArcGISx6410.1/python.exe because that seems to be the problem...

Thanks,
john

[KevinHibma]

I'm not entirely certain of your setup/configuration, but I can say this: inside ArcMap, Python is 32bit.
I'm not really sure how you're seeing that error from the Python window. I tried changing a few settings on my machine to my 64bit Python and my PyWindow continues to work fine.

I dont know much about IronPy and the settings you've shown in regards to that - I dont imagine that its a problem though.
The only thing strange about the messages you've posted are:

File "C:\Python27\Lib\site-packages\numpy\lib\type_check.py", line 8, in <module>
    import numpy.core.numeric as _nx
  File "C:\Python27\Lib\site-packages\numpy\core\__init__.py", line 5, in <module>
    import multiarray

The directory is pointing at c:\Python27\lib
My guess here is either you've installed your own Python27 prior to installing Desktop (as well as matplotlib and numpy) or did you just change the directory it installs to while installing Destkop?
If the first (you installed it) - is it possible there isn't a complete link between Desktop and Python and (again I cant imagine how) Desktop is now falling through to the only Python it can find - the 64bit one?

Just checking: everything inside Desktop/Python window works fine before installing 64bit Background Processing?

[ChrisSnyder]

That error message (ImportError: DLL load failed: %1 is not a valid Win32 application) sounds familiar: http://forums.arcgis.com/threads/78704-Python-subprocess-module-and-64-bit-background-geoprocessing-...

I fixed my issue (which may be a bit different from yours) by permanently including a 'PATH' envr variable pointing to the 64-bit Python directory (Note that PATH does not include a ref to the 32-bit version directory). My Python window in ArcMap loads arcpy (32-bit) just fine... but I never noticed that it didn't.

[JohnBurkhart1]

Thanks for the responses.

I do certainly have Python installed natively... and use it heavily. My environment variables are as follows:

PATH=C:\Python27;C:\Python27\ArcGISx6410.1;C:\Python27\Scripts;C:\Python27\Lib\site-packages\django\bin
PYTHONPATH=C:\Python27\Lib\;C:\Python27\Lib\site-packages\;C:\pythonnet
pythonExePath=C:\Python27\ArcGISx6410.1\python.exe

Everything 'arcpy' and python related work fine if I work from Ipython **outside** of ArcMap. It is only the python window inside arcmap that fails. I've posted the full traceback below, but the problem is clear -- it is trying to load 64bit DLLs. But how do I change the behavior of this python instance inside ArcMap?? Is this IronPython?? I shouldn't have to change environment variables (that will break other things) just to get python to work inside ESRI products. (though I did try and it still did not work)

If I look at the sys.path inside ArcMap python it is clear it is reading my env variables (e.g. C:\pythonnet), but it is also adding it's own. It doesn't seem to be reading the PATH variable at all. It is strange that it inserted the arcpy path in position 1, ahead of pythonnet, but after python27/lib.

>>> sys.path
['C:\\Python27\\Lib', u'c:\\program files (x86)\\arcgis\\desktop10.1\\arcpy', 'C:\\Python27\\Lib\\site-packages', 'C:\\pythonnet', 'C:\\windows\\system32\\python27.zip', 'C:\\Python27\\ArcGIS10.1\\Lib', 'C:\\Python27\\ArcGIS10.1\\DLLs', 'C:\\Python27\\ArcGIS10.1\\Lib\\lib-tk', 'C:\\ProgramData\\Microsoft\\Windows\\Start Menu\\Programs\\ArcGIS', 'C:\\Program Files (x86)\\ArcGIS\\Desktop10.1\\bin', 'C:\\Python27\\ArcGIS10.1', 'C:\\Python27\\ArcGIS10.1\\lib\\site-packages', 'C:\\Program Files (x86)\\ArcGIS\\Desktop10.1\\arcpy', 'C:\\Program Files (x86)\\ArcGIS\\Desktop10.1\\ArcToolbox\\Scripts']
>>> 

So again, I'm stumped... insights welcome.

--john

FULL Traceback:

Runtime error 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "\\energycorp.com\Statkraft\users\Users01\u35888\ArcGIS\arc_startup.py", line 2, in <module>
    import arcpy
  File "c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\__init__.py", line 24, in <module>
    from arcpy.toolbox import *
  File "c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\toolbox.py", line 342, in <module>
    from management import Graph, GraphTemplate
  File "c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\management.py", line 22, in <module>
    import _management
  File "c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\_management.py", line 14, in <module>
    import _graph
  File "c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\_graph.py", line 27, in <module>
    import numpy
  File "C:\Python27\Lib\site-packages\numpy\__init__.py", line 147, in <module>
    import add_newdocs
  File "C:\Python27\Lib\site-packages\numpy\add_newdocs.py", line 9, in <module>
    from numpy.lib import add_newdoc
  File "C:\Python27\Lib\site-packages\numpy\lib\__init__.py", line 4, in <module>
    from type_check import *
  File "C:\Python27\Lib\site-packages\numpy\lib\type_check.py", line 8, in <module>
    import numpy.core.numeric as _nx
  File "C:\Python27\Lib\site-packages\numpy\core\__init__.py", line 5, in <module>
    import multiarray
ImportError: DLL load failed: %1 is not a valid Win32 application.
>>> 

[KevinHibma]

John,

I just came across this thread, maybe give it a read: http://forums.arcgis.com/threads/75553-SOLVED-Error-8273-geoprocessing-services-crashing-on-SOM-star...
In short > check your registry to see where your system "thinks" Python lives...

[ChrisSnyder]

I've never used Iron Python, but that may be a likely reason things are going wrong for you.

For what it's worth, my PATH variables/values:

PATH = c:\oracle\product\11.2.0\client_1\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Python27\ArcGISx6410.1

It's interesting to note that my PATH (but only after I edited it to include a ref to C:\Python27\ArcGISx6410.1) is the ONLY envr system varaiable that makes reference to anything Python. I assume everything else is handled in the registry... My machine is only a few months old, and has only had ArcGIS v10.1 and Python2.7 installed on it (no other older versions). My other machine (that is several years old and had several versions of ArcGIS /Python on it) has tons of refernces to Python in the system envr variables. Hmmm...

[JohnBurkhart1]

I'm still having no luck with this situation. I would appreciate someone from ESRI provide some more input. I'm asking about the "console python", meaning the python window that is launched from *within* Arcmap. I have no problems when running stand-alone scripts outside of Arcmap.

I am also having problems with Addins, and I fear that this problem is causing those to fail as well.

Could you please tell me:

1) What python.exe runs in the console window?
2) What system environment variables are required by the console python?
3) what system environment variables are checked by the console python?
4) Is it possible to make ESRI Addins use the system python and not the console python?

Thank you,
john

[JohnBurkhart1 · Accepted solution]

I would be extremely grateful for some input from ESRI to know whether my solution is 'normal'. But in the end, this seems to be what was required to get things working:

In order to get ESRI to run, and to use 64 bit processing:

1) Download and run the 64bit background geoprocessing tools:
Luckily I found it once! I can't seem to find the link again...

2) Add the following to the system PYTHONPATH:
;C:\Python27\ArcGISx6410.1\DLLs;C:\Python27\ArcGISx6410.1\lib;C:\Python27\ArcGISx6410.1\lib\plat-win;C:\Python27\ArcGISx6410.1\lib\lib-tk;C:\Python27\ArcGISx6410.1;C:\Program Files (x86)\ArcGIS\Desktop10.1\bin64;C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy;C:\Program Files (x86)\ArcGIS\Desktop10.1\ArcToolbox\Scripts

3) Create a system environment variable PYTHONSTARTUP and point to a python script for Arcmap Python so it doesn't get confused with 64 bit imports from the PYTHONPATH:

# My startup file  import sys print 'using ', file if sys.executable in ['C:\\Python27\\ArcGIS10.1\\python.exe', 'C:\\Program Files (x86)\\ArcGIS\\Desktop10.1\\bin\\ArcMap.exe']:     print('Resetting Path')     sys.path = [             #python32 for ESRI             'C:\\windows\\system32\\python27.zip',             'C:\\Python27\\ArcGIS10.1\\DLLs',             'C:\\Python27\\ArcGIS10.1\\lib',             'C:\\Python27\\ArcGIS10.1\\lib\\plat-win',             'C:\\Python27\\ArcGIS10.1\\lib\\lib-tk',             'C:\\Python27\\ArcGIS10.1',             'C:\\Python27\\ArcGIS10.1\\lib\\site-packages',             'C:\\Program Files (x86)\\ArcGIS\\Desktop10.1\\bin',             'C:\\Program Files (x86)\\ArcGIS\\Desktop10.1\\arcpy',             'C:\\Program Files (x86)\\ArcGIS\\Desktop10.1\\ArcToolbox\\Scripts',             #defaults             'C:\\Python27\\Lib\\',             'C:\\Python27\\Lib\\site-packages',             'C:\\pythonnet'             ]      import arcpy      from arcpy import env     import numpy      import time      env.overwriteOutput = True

If this is really the 'proper' way of doing things, some documentation would be nice... the help docs seem to not have very much information on any of this! I found better documentation online for open source geoprocessing tools.

[MelanieSummers]

Hi John,

Do you have the 64-bit background processing patch installed? If not, you may want to install it, you can read more about running scripts using 64 bit-processing here.

Best,
Melanie S.

[KevinHibma]

I want to say the steps you've taken are not normal.
On a brand new machine after you install Desktop, and 64-bit Background, thats it. No environment variables, no Python path settings, nothing like you've needed to do.

For me to document a "fix" (like in your case), I'd need to be able to reproduce the problem so I can fully understand it.
I still haven't found a way to "break" the Python window by having 64-bit Background and changing a combination of environment variables.

I'm glad you figured it out. Your well documented answer is good in case someone else encounters this situation. I'll spend a little bit more time trying to reproduce, but like I said, so far I've been unsuccessful - it all just works.