connecting ArcPy and CUDA

[MikeTischler]

I've been working on creating a geoprocessing environment that links ArcPy with NVIDIA CUDA GPU capabilities for the past couple weeks, and didn't want my lessons learned to go to waste.  I'm really enjoying development in python in ArcGIS 10, and I came across the PyCUDA library which allows python integration into the CUDA API.  This takes a lot of patience, and still requires some serious development to create GPU code.  But, for those who like a challenge, here's how I got things to work.

My system specs, the important parts:
Windows Server 2008 rc2 x64
TESLA S2050 GPU
64bit TESLA Driver
64bit CUDA Toolkit
Microsoft Visual Studio 2008 64bit (the C++ parts especially)

There are several issues that needed to be resolved, mostly dealing with the 32bit/64bit divide.  ArcGIS, and therefore ArcPy, are 32bit programs only, but need to communicate with 64bit CUDA drivers and toolkits, on a 64bit OS.  (Note: If you have a different configuration, you might be able to use one of the precompiled PyCuda binaries available here http://www.lfd.uci.edu/~gohlke/pythonlibs/#pycuda

To start, I installed in this order

1.  Tesla Driver 263.06 winserv2008 rc2
2.  CUDA Toolkit 3.2.16 win64
3.  ArcGIS (and Python 2.6.5, 32bit)

now comes the tricky part.  PyCuda (http://mathema.tician.de/software/pycuda) relies on a subset of a series of C++ libraries, the whole of which is called Boost.  Specifically, PyCuda needs 3 boost libraries...date_time, python, and thread.  In order to get everything to be friendly, I had to compile both PyCuda and the necessary boost libraries from source. 

So...where to start?

1.  Download Boost source  http://sourceforge.net/projects/boost/files/boost/1.46.1/boost_1_46_1.zip/download
2.  Download PyCuda source http://pypi.python.org/packages/source/p/pycuda/pycuda-0.94.2.tar.gz#md5=d4651a015f1cc2b5702d969ff15...

Let's say you're putting these in C:\working .  I created a directory called boost_src into which I copied and then unzipped the boost source code.  I extracted the PyCuda source into pycuda-0.94-2, which is the default extract directory.

Open a command line prompt and

cd c:\working\boost_src\boost_1_46_1

then

bootstrap

bootstrap will create the bjam.exe program, which is a compiler for the boost libraries.  Now, if you were to build and compile ALL the boost libraries, it would take quite a while.  So, we need to do just the few we're interested in: date_time, python, and thread.

from the same command line prompt:

/bjam python toolset=msvc link=shared threading=multi runtime-link=shared architecture=x86 address-model=32 stage

don't forget the slash at the beginning.  This tells the boost compiler to build .dll's that will work across the 32bit/64bit divide.

This command will take a few minutes.  The compiler is building not only the boost.python library (designated by the 'python' argument in the command), but the necessary dependency libraries.  Building the next two libraries will be quicker, as many of the dependency libraries are the same as the first.

To build the 2nd and 3rd libraries, issue the same command, changing 'python' to 'date_time' and hit enter.  When that's done, change it one more time to 'thread' and hit enter.  That will create the necessary .dll's in the /stage directory in your boost root directory.

Ok - now to build PyCuda.  At this point, you may need to create an environmental variable called "PATH" (right click Computer, Properties,Advanced System Settings,Environment Variables, New) and add C:\Python26\ArcGIS10.0.  This will tell windows where to find the python.exe file.

From the command line, :

cd c:\working\pycuda-0.94.2

and then:

python configure.py

this will configure PyCuda for building and install.  A configuration file that can be opened and edited in wordpad is created called siteconf.py.  Several things will need to be edited here.  A working siteconf.py file may look like:

BOOST_INC_DIR = ['C:/working/boost_src/boost_1_46_1']
BOOST_LIB_DIR = ['C:/working/boost_src/boost_1_46_1/stage/lib']
BOOST_COMPILER = 'msvc9'
USE_SHIPPED_BOOST = False
BOOST_PYTHON_LIBNAME = ['boost_python-vc90-mt-1_46_1']
BOOST_THREAD_LIBNAME = ['boost_thread-vc90-mt-1_46_1']
CUDA_TRACE = False
CUDA_ENABLE_GL = False
CUDA_ROOT = 'C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v3.2'
CUDADRV_LIB_DIR = ['C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v3.2/lib/Win32']
CUDADRV_LIBNAME = ['cuda']
CXXFLAGS = ['/EHsc','/DBOOST_ALL_NO_LIB']
LDFLAGS = ['/FORCE']

This will tell pycuda where to find the right boost libraries, which compiler to use, as well as the CUDA toolkit and driver.

Now, you should be able to build and install pycuda.  Again, from the command line:

python setup.py build

python setup.py install

If all goes well, you should get a new directory in your C:\Python26\ArcGIS10.0\Lib\site-packages directory that's something like pycuda-0.94.2-py2.6-win32.egg.  There are a few things that SHOULD happen here automatically, but may not.  For example, the python libraries 'setuptools' and 'pytools' should automatically be downloaded, installed, and used for the PyCuda install. 

Now - you can check to see if things are working ok so far by issuing a few python commands.  Open a new command prompt, and make sure you're not in the pycuda directory where you issued the install command. 

python

to open a python prompt

import pycuda.autoinit

pycuda.autoinit is the initialization procedure for PyCuda that connects to the GPU.  This is the most basic, most fundamental command and is usually the first command issued.  If you don't get any errors, GREAT!  You're almost home.

The last part deals with actually writing and compiling scripts.  PyCuda will work with the NVIDIA GPU compiler program 'nvcc' to compile code to run on the GPU.  But, nvcc has to be told to compile 32bit code, since we'll be using 32bit python to leverage ArcPy. 

Open C:\Python26\ArcGIS10.0\Lib\site-packages\pycuda-0.94.2-py2.6-win32.egg\pycuda\compiler.py and go all the way to the bottom.  You'll see a class called SourceModule that starts like:

class SourceModule(object):
    def __init__(self, source, nvcc="nvcc", options=[], keep=False,
            no_extern_c=False, arch=None, code=None, cache_dir=None,
            include_dirs=[]):

You need to change the

options[]

to look like

options=['-m32']

(the PyCuda development team is aware of this bug, which will be fixed in upcoming versions of PyCuda)

This tells nvcc to compile 32bit code.  Now, you should be able to run some test scripts using pycuda, similar to the one on the PyCuda documentation website http://documen.tician.de/pycuda/

This worked for me, and it took me quite a while to figure out how to glue everything together.  But, now I am able to leverage all the ArcPy geoprocessing tasks with my custom GPU kernels. 

Now, don't get the wrong idea - you WILL NOT be able to run ArcGIS tools on a GPU.  That's up to ESRI.  But, this will allow you to build geoprocessing tasks that can be extended to work on a GPU.

A few more things.  You will likely have to add more directories to your PATH environment variable.  Mine looks like this

C:\Python26\ArcGIS10.0;
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC;
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin;
C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE;
C:\working\boost_src\boost_1_46_1\stage\lib;
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\include

 

Also, you'll see a few system variables (below the environment variables) related to the CUDA toolkit.  Specifically, you'll probably need to change CUDA_LIB_PATH to %CUDA_PATH%\lib\Win32 instead of %CUDA_PATH%\lib\x64.

Hope this helps!
Mike

[JasonScheirer]

Thanks for this post, I'm sure this information will come in handy for quite a few people. For those wishing to get started with less hassle, there are pre-built binaries already available at http://www.lfd.uci.edu/~gohlke/pythonlibs/#pycuda

[MikeTischler]

Thanks for this post, I'm sure this information will come in handy for quite a few people. For those wishing to get started with less hassle, there are pre-built binaries already available at http://www.lfd.uci.edu/~gohlke/pythonlibs/#pycuda

Yep - I referenced that link up top.  In fact, when I had a purely 32bit configuration on another machine, the precompiled binary from that site worked great.

[DINGXIANFENG]

Thanks for this helpful post. but is there any examples to run a geoprocessing in this ArcPy && CUDA environment ?

[MikeTischler]

Goodmanding - I've been working on an application that does preprocessing via ArcPy and additional geoprocessing through custom non-arcpy code via GPU, but it's not very solid yet.  Also, keep in mind that any gpu solution will change based on the GPU hardware (this is much different than CPU computing) and data configuration of your geoprocessing problem.