using mutiprocessing module - can't find python.exe anymore?

[ChrisSnyder]

I finally had a good reason to start writing an arcpy script that uses the multiprocessing module (new in v2.6). Great! I start coding following this example: http://pythongisandstuff.wordpress.com/2011/07/11/using-arcpy-with-multiprocessing-%E2%80%93-part-3/

When I run my script it throws this weird error when it gets to the multiprocessing part:

A new PythonWin window pops up with an error "Could not load the file from C:\Python26\ArcGIS10.0\Lib\site-packages\pythonwin"

Oh crap...

I go to a coworkers machine and try to run my script.... Same error!!!

After a long while I deduce that for whatever reason the mutiprocessing module can't find python.exe anymore. I debug it untill I finally get to the problem: line 179 in the "forking.py" script (C:\Python26\ArcGIS10.0\Lib\multiprocessing\Forking.py). Here is that code:

if sys.executable.lower().endswith('pythonservice.exe'):
   _python_exe = os.path.join(sys.exec_prefix, 'python.exe')
else:
   _python_exe = sys.executable

Problem is that the executable I am using DOES NOT end with "pythonservice.exe" (cause I am using PythonWin.exe!), so it reverts to using whatever the sys.executable variable points to (which is pythonwin.exe). Ah ah!

So, of course when I hardcode the offending lines of code to be:

if sys.executable.lower().endswith('pythonservice.exe'):
   _python_exe = os.path.join(sys.exec_prefix, 'python.exe')
else:
   _python_exe =  os.path.join(sys.exec_prefix, 'python.exe')

Everything works fine!!! AWESOME!

So my questons are:

1. If my coworker and I are having this issue, is anyone else have this issue?
2. Why can't the multiprocessing module find python.exe? Could it be related to arcpy "taking over" the Python install directory???
3. Other than hardcoding the correct path to python.exe in the "forking.py" script, what can I do to make a permanent fix for this?

P.S. Setting the PYTHONPATH system variable doesn't do the trick...

....Also, running ArcGIS v10.0 SP2, Windows7

Any ideas???

[DanPatterson_Retired]

Chris
can't help it ... are you sure that "forking.py" is spelled correctly 🙂

[ChrisSnyder]

Yes, forking with python is pretty interesting...

I though the comment in the f{o,u}rking script on line 174-177 was kinda fishy:

#
# _python_exe is the assumed path to the python executable.
# People embedding Python want to modify it.
#

[DanPatterson_Retired]

😄
best of luck... been playing with recarrays and I been having difficulty with spelling as of late

[JasonScheirer]

You may want to do it  the documented way.

[ChrisSnyder]

...docume-what?

multiprocessing.set_executable()

A million thanks Jason!!!

[ChrisSnyder]

BTW: Jason, do you/ESRI have any multiprocessing/subprocess/parallelpython examples using arcpy/gp? More examples, the better... I've seen only a handfull of examples for multiprocessing with arcypy (well actually only one).

Also, Do you personally (so as not to confuse it with your employer) have any recomendations about the "best" method use for parallel processing in Python using ESRI objects??? I'm trying to modernize my parallel scripts that use the (antiquated?) os.spawnv, and am seeing a lot of different options such as parallel python, multiprocessing, subprocess, etc.

Any recomendations?

[JasonScheirer]

I think a couple are due up on the geoprocessing resource center in the coming weeks, if I'm not mistaken.

[ChrisSnyder]

Cool - looking forward to seeing those...

BTW: WOW! Looks like you can now finally process Spatial Analyst tools in parallel now in v10/arcpy. Regardless it took me a year to realize it, that little discovery this morning made my day (probably my year actually!). Never noticed that (***very important to me***) tidbit in any of the "what's new in v10" documents... Man, I would have totally gone to arcpy right away (like a fly to sh*%!) had I noticed that sooner!!!

Anyway - whoohoo! 🙂