SciPy.ndimage and ArcGIS Tools

TobiasLitherland · ‎07-16-2015

I have just discovered the scipy.ndimage.label method, which is just amazing at simple raster object detection.

I have made a script that uses this method extensively, and would like to create a script tool in an arcgis toolbox.

The problem is that I am not able to get a working install of scipy with native arcgis python.

My solution has been to install Anaconda Python in parallell with ArcGIS Python, and then make the tool run the script via the command line using the Anaconda Python passing the GetParameterAsText's as command line arguments:

os.system('C:\...\Anaconda\python.exe "C:\GIS_Scripts\...\My_script.py" "arg1" "arg2" "arg3"')

This works, but it lacks proper error propagation and messages to the tool window in ArcMap.

So, two questions:

1) Have any of you managed to use SciPy with native ArcGIS python, and if you have, can you make scipy.ndimage work? I seem to constantly be missing some module references.

2) Is there any way to decouple the native Python install, and make ArcGIS use another python executable but while still making arcpy available?

Kind regards,

Tobias

ShaunWalbridge · ‎07-16-2015

Hello Tobias Litherland (and thanks for the heads-up Xander Bakker):

If you have Pro installed, that's the easiest route -- it ships with SciPy, and is usable today. Further improvements are coming soon to our other products, but that doesn't help you for solving your immediate needs. For that, you can try using Python path files (.pth) to bridge between the environments.

WARNING: This approach is risky, and can potentially break your Python installation, so be careful, and this isn't a directly supported method. Because you're using different versions of packages like NumPy under the hood, strange things can happen, and sometimes introduce crashes.

1. Find the site packages path for your Anaconda install. On my machine, it was C:\Miniconda\envs\default\Lib\site-packages. Copy that path into a file called anaconda.pth.

2. Copy anaconda.pth into C:\Python27\ArcGIS10.3\Lib\site-packages (or the correct location if you used a different installation path).

3. Open Python, type:

import sys
print(sys.path)

If it worked correctly, you should see a new entry for the Anaconda environment from within the ArcGIS installed version of Python, and import scipy should work. Again, note that this may cause other problems, and if anything is breaking, delete the pth file to return your setup to a stock install.

Cheers,

Shaun

View solution in original post

XanderBakker · ‎07-16-2015

Hi Tobias Litherland ,

Sounds great what you are working on. The first question that comes into mind is the version of ArcGIS that you are using. Esri has announced the integration of SciPy and ArcGIS (see Esri Advances Scientific Analysis with SciPy ) and is investing a lot of effort to offer more functionality and integration with the scientific community (see this white paper, July 2015): http://www.esri.com/library/whitepapers/pdfs/esri-and-the-scientific-community.pdf

I'm going to mention Shaun Walbridge and Kevin Butler who gave a great presentation at the DevSummit 2015 (https://4326.us/scipy/#/ ) about ArcGIS and working with Scientific Data the SciPy Stack, to see if they are listening and can help you further.

You may also be interested in this blog post: Esri and the Science Community: The Year Ahead | ArcGIS Blog

Kind regards, Xander

TobiasLitherland · ‎07-16-2015

Thanks for all the info, Xander Bakker!

I am looking forward to more integration of SciPy in ArcGIS, as the speed and functionality of some of the methods in the module just blows Spatial Analyst out of the water (I'm sorry to say!).

I found the white papers and the power points from the DevSummit presentations, but I did not seem to find any substantial information on ha to actually make the bridge work.

I am using ArcGIS Desktop 10.3 Advanced.

ShaunWalbridge · ‎07-16-2015

Hello Tobias Litherland (and thanks for the heads-up Xander Bakker):

If you have Pro installed, that's the easiest route -- it ships with SciPy, and is usable today. Further improvements are coming soon to our other products, but that doesn't help you for solving your immediate needs. For that, you can try using Python path files (.pth) to bridge between the environments.

WARNING: This approach is risky, and can potentially break your Python installation, so be careful, and this isn't a directly supported method. Because you're using different versions of packages like NumPy under the hood, strange things can happen, and sometimes introduce crashes.

1. Find the site packages path for your Anaconda install. On my machine, it was C:\Miniconda\envs\default\Lib\site-packages. Copy that path into a file called anaconda.pth.

2. Copy anaconda.pth into C:\Python27\ArcGIS10.3\Lib\site-packages (or the correct location if you used a different installation path).

3. Open Python, type:

import sys
print(sys.path)

If it worked correctly, you should see a new entry for the Anaconda environment from within the ArcGIS installed version of Python, and import scipy should work. Again, note that this may cause other problems, and if anything is breaking, delete the pth file to return your setup to a stock install.

Cheers,

Shaun

TobiasLitherland · ‎07-16-2015

Hi Shaun Walbridge,

Thanks for the heads-up!

I generally do most of my work in scripts via Anaconda, and my Anaconda install references the arcpy site package via a .pth-file. I hoped that the same would work the other way, and had tried it, but it seems like something is missing.

I added C:\...\Anaconda\Lib\site-packages to anaconda.pth and placed this in the C:\Python27\ArcGIS10.3\Lib\site-packages. I was then able to import scipy but if i attempt import scipy.ndimage it failed:

>>> import scipy

>>> import scipy.ndimage

Runtime error

Traceback (most recent call last):

File "<string>", line 1, in <module>

File "C:\Users\u34386\AppData\Local\Continuum\Anaconda\Lib\site-packages\scipy\ndimage\__init__.py", line 172, in <module>

from .filters import *

File "C:\Users\u34386\AppData\Local\Continuum\Anaconda\Lib\site-packages\scipy\ndimage\filters.py", line 36, in <module>

from . import _nd_image

ImportError: numpy.core.multiarray failed to import

I assume that this is the mismatch in numpy-versions you mentioned. It would explain why the scipy.ndimage import failes at least, but at the same time it is strange that scipy seems to work (apparently).

We have access to Pro, so that might actually be a solution. I'll check it out. Thanks!

ShaunWalbridge · ‎07-17-2015

Yeah, that's what I'd expect out of the box with Conda -- the version of NumPy that is on your sys.path is 1.7.1, and the version in Conda is 1.9.1 -- NumPy is not ABI compatible between major releases. If you want to stick to ArcMap, you can try making a environment with NumPy 1.7.1 and a related SciPy release, e.g.:

conda install numpy=1.7.1conda install scipy=0.13.2

And see if it works linked against that conda environment. Good luck!

TobiasLitherland · ‎09-09-2015

Hei again Shaun Walbridge!

I just got ArcGIS Pro up and running, and with only minor alterations to the code I got i working seamlessly with ArcGIS Python 3.4. The SciPy-version packaged with ArcGIS Pro has all the tools in I need for the job.

Great stuff, and a nice entry point for me to slowly convert to ArcGIS Pro.

ShaunWalbridge · ‎09-09-2015

Great! Glad to hear it. The 10.4 release will include the same packages currently included in Pro, so you should be able to write code that will work equally well in both ArcMap and Pro at that point. Let us know if you have any further issues, and look forward to hear more about your work with raster object detection.

Cheers,

Shaun