10.1 SP1 64bit geoprocessing, error importing arcpy

[JeffKruys]

This error only occurs when using the 64bit python.exe that is installed with the SP1 for 64bit geoprocessing. And it only occurs when importing the arcpy module in a python script that is called from within another python script. I am using Windows 2008 Server R2 and ArcGIS Desktop 10.1 and the Python 2.7 that is installed with it.

Here's script 1, named test.py:

import arcpy, os, sys print "Imported arcpy, os and sys modules successfully in main script" os.system(sys.argv[0].replace('.py','_slave.py'))

Here's script 2, named test_slave.py:

import arcpy print "Imported arcpy module successfully in slave script"

I'm initiating the test.py script by entering "X:\fullpath\test.py" at the Windows command prompt.

In the past, in 32bit before installing SP1, this runs with no problem. Now, after installing SP1 and switching the Windows association with the .py extension to the 64bit python.exe, this is what I get:

Imported arcpy, os and sys modules successfully in main script Traceback (most recent call last):   File "X:\fullpath\test_slave.py", line 1, in <module>     import arcpy   File "c:\program files (x86)\arcgis\desktop10.1\ArcPy\arcpy\__init__.py", line 21, in <module>     from arcpy.geoprocessing import gp   File "c:\program files (x86)\arcgis\desktop10.1\ArcPy\arcpy\geoprocessing\__init__.py", line 14, in <module>     from _base import *   File "c:\program files (x86)\arcgis\desktop10.1\ArcPy\arcpy\geoprocessing\_base.py", line 14, in <module>     import arcgisscripting ImportError: DLL load failed: %1 is not a valid Win32 application.

And in fact, when I switch the Windows association with the .py extension back to the 32bit python.exe, I continue to get this error, until I uninstall SP1 for 64bit geoprocessing.

Is this a bug, or is there some workaround I can employ to get this working under 64bit? (Please don't recommend that I rewrite all my scripts to not call scripts from within scripts!)

[T__WayneWhitley]

I believe your problem stems from your 'os.system' call which runs your 'test_slave.py' in a subshell using the standard C function, 'system()'.  See:

http://docs.python.org/2/library/os.html#os.system

...where I quote:

"The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function."


A similar error is noted at this forum post, see:

http://forums.arcgis.com/threads/66847-32-bit-and-64-bit-Python-2.7-on-the-same-machine

As you can see, although using slightly different techniques, you may call the error precisely identical in nature.  It is the same error tripped by running in a shell process, a 32-bit 'operating process space', for lack of a better term.  This was your critical clue, your error:

ImportError: DLL load failed: %1 is not a valid Win32 application.


If you want to run this way, try the suggestion by Chris Snyder, using the subprocess module, see:

http://forums.arcgis.com/threads/75207-Excel-and-Python-Friends-or-Foes?#20


Chris lists the command there as in that noted in italics below.  If you will notice the use of the pythonExePath, that should be set as your Python executable path, which I believe you could set to the 64-bit version (the post at the link above concerned 32-bit but I think you should be able to to launch either 32-bit/64-bit python.exe).

pythonExePath = r'C:\Python27\ArcGISx6410.1\pythonw.exe'
subprocess.Popen([pythonExePath, childScriptPath, str(inputVar1), str(inputVar2)], shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)


Hope that helps to clarify.

Enjoy,
Wayne


ADDITIONAL NOTE:
'pythonw.exe' or 'python.exe', what is the difference?  Use the one with 'w' appended to 'python' to not display the command window...i.e., 'python' + 'w' + '.exe', or 'pythonw.exe'.

"This suppresses the terminal window on startup."
http://docs.python.org/2/using/windows.html#executing-scripts

[JeffKruys]

Thanks for the response. Here's my latest test using the subprocess module, which still fails if arcpy is imported in both the main script and in the subprocess script. Along the way, I discovered that "sys.executable" returns the full path of the python.exe currently being executed.

test.py:

import subprocess, sys, os, arcpy
cmd = sys.executable + " " + sys.argv[0].replace('.py','_slave.py')
print "    - About to run: " + cmd
theprocess = subprocess.Popen(cmd, shell=False)
theprocess.communicate()
print "    - All done"

test_slave.py:

import sys, time
print "        - Currently executing: " + sys.executable + " " + sys.argv[0]
print "        - sleeping for 5 sec"
time.sleep(5)
print "        - importing arcpy"
import arcpy
print "        - success; returning to main script"

Result:

    - About to run: C:\Python27\ArcGISx6410.1\python.exe X:\fullpath\test_slave.py
        - Currently executing: C:\Python27\ArcGISx6410.1\python.exe X:\fullpath\test_slave.py
        - sleeping for 5 sec
        - importing arcpy
Traceback (most recent call last):
  File "X:\fullpath\test_slave.py", line 6, in <module>
    import arcpy
  File "c:\program files (x86)\arcgis\desktop10.1\ArcPy\arcpy\__init__.py", line 21, in <module>
    from arcpy.geoprocessing import gp
  File "c:\program files (x86)\arcgis\desktop10.1\ArcPy\arcpy\geoprocessing\__init__.py", line 14, in <module>
    from _base import *
  File "c:\program files (x86)\arcgis\desktop10.1\ArcPy\arcpy\geoprocessing\_base.py", line 14, in <module>
    import arcgisscripting
ImportError: DLL load failed: %1 is not a valid Win32 application.
    - All done

Any ideas? Thanks!

[T__WayneWhitley]

You are making the same critical error....maybe you aren't understanding.  sys.executable isn't going to cut it...

Try this, maybe you will see -- open IDLE if you have it, and simply issue this command (after importing sys of course):

>>> sys.executable


What do you get fed back to you?  I'll bet you did not get the correct path, which should have been this:

'C:\Python27\ArcGISx6410.1\pythonw.exe'


Did I guess correctly??

-Wayne


EDIT:  I am adding to my post because I hope you do not uninstall anything - it is going to be just fine with 2 versions of the python.exe (32 bit/64 bit versions)......the part you are not quite grasping is that you can EXPLICITLY state the pythonExePath variable as in post #2.  That, I am confident, is your ticket.

(...and thanks goes again to Chris Snyder for his contributions [see the links provided] to help get the concept through my thick head!  Also Mr. Hibma at ESRI, etc., the list goes on...)

[JeffKruys]

You're right, there's definitely something I'm not getting. In IDLE, I do get 'C:\\Python27\\ArcGISx6410.1\\pythonw.exe', and I'm not sure what that proves...

Okay, here's another version of test.py:

import subprocess, sys, os, arcpy
cmd = r'C:\Python27\ArcGISx6410.1\python.exe ' + sys.argv[0].replace('.py','_slave.py')
print "    - About to run: " + cmd
theprocess = subprocess.Popen(cmd, shell=False)
theprocess.communicate()
print "All done"

I get the same error with this. If I specify pythonw.exe on line 2, I get the same failure, but no indication of it because the output is being sent out to oblivion. Anyway, I would like not to have to hard-code the path to python.exe because the scripts I'm writing will be run by other folks on other machines.

[JeffKruys · Accepted solution]

All fixed now. I was providing a single string (the command as I would enter it at the command line) as the first argument to subprocess.Popen, but it wants a list (as you had specified in your first response). This version of test.py works fine:

import subprocess, sys, os, arcpy print "    - About to run: " + sys.executable + " " + sys.argv[0].replace('.py','_slave.py') theprocess = subprocess.Popen([sys.executable,sys.argv[0].replace('.py','_slave.py')], shell=False) theprocess.communicate() print "    - All done"

I don't know why specifying a single command string instead of a list would work at all, invoking the slave script and only failing when trying to import arcpy a second time. But the above works, so I'm happy. Thanks a lot for your help.

[T__WayneWhitley]

Beautiful, glad it is now running - I knew you'd get it eventually.  About the 'python' vs 'pythonw' convention - no msgs are being sent into 'oblivion' but by design, no messages are being sent, i.e. they are suppressed.

If you would please, go ahead and mark this post 'answered' using the controls to the right corner...there's a green checkmark.
That will definitely aid others just looking directly for the marked answer.


Enjoy,
Wayne

EDIT: Notice that the list fed into subprocess takes on the same command line you would [very likely] enter for scheduled tasks:

(the exe path), (the script file), (param1), (param2), param3), ...etc.

It is a whole new process that you are making your 'master' script aware of, although after it is started, the 'subprocess' is more or less autonomous.  So you need the whole command string.  The other route you were heading down does not function the same way, but calls the subshell C by default and there is no opting out of that....  There's a lot more to explain about that, but I am not even going down that rabbit hole!  It is just important to know you have to be specific about which 'process space' you want to use, and using os.system calls the same C function system() does and runs in 32-bit space....so when you get '...not a valid Win32 application', it is speaking of your 64-bit exe.  The C process is 'unaware' of what your 64-bit py even is.

I hope I described that properly enough.