topic Re: Should Esri ship third-party modules (like scipy) with ArcGIS? in Python Questions

Should Esri ship third-party modules (like scipy) with ArcGIS?

curtvprice — Wed, 08 Aug 2012 17:52:37 GMT

The ArcGIS Python 2.7 distribution for 10.1 includes (in addition to the python standard library) the user-contributed modules numpy and matplotlib. It would be really great to include some more modules. If you agree, vote this item up and add your favorites (and use cases) in the comments.

Vote here, and also feel free to vote the idea up:

ideas.esri.com: Ship additional python extensions

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

MathewCoyle — Wed, 08 Aug 2012 18:42:20 GMT

pywin32 specifically the win32api.

I like to use it for quasi OS authentication by accessing active directory user names and permissions. As a custom install right now it is a hassle to package it with the standard deployment.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

JasonScheirer — Wed, 08 Aug 2012 20:00:07 GMT

I will be visiting this thread fairly often to keep up-to-date on what you all think. We on the Python team want you to be happy and producting using Python on the ArcGIS stack.

Please keep in mind that we can't include everything requested as there is a lot of work in testing and integration for each library ([Windows|Linux]x[32 bit|64 bit]x[Many other factors]). For example, I had to help with a workaround for a memory leak in matplotlib that was causing many spatial statistics tools to leak huge amounts of memory. And what would pywin32 look like on Linux? Adding a third-party module means adding an additional dependency in ArcGIS on that third-party which may or may not update regularly or anywhere in sync with our own timetables. Our release dates and the release dates of the third party can result in scares for us: numpy didn't have support for Python 2.7 until a week before 10.1 beta's cutoff date, so we almost had to stick with 2.6 for 10.1. I can only imagine how much closer (or over?) we would have run if we had a scipy dependency on top of that. And the opposite is true, too: we started with upgrading to 2.7 in our daily builds when it was first released, updated and tested 10.1 to Python 2.7.2 during the development cycle and then 2.7.3 came out right before 10.1's release date. I can see one or two libraries being frozen at a version you don't like (maybe they're crashy or have a bug that only existed for a few builds or have awkward APIs which are later fixed) on every install of ArcGIS, and fear that could cause more pain than the hassle of shipping your own with your tools. Not to mention the entirely unknown quantity of not just porting ourselves to Python 3, but waiting for all our third-party dependencies to port as well.

There's also reduplication of functionality in a lot of these libraries. I agree that we should get our GDAL bindings actually shipping and not just on the resource center. But on the other side, there's no realistic use case for Shapely considering ArcGIS already handles geometry in a very flexible, intelligent way. The only missing part is that it needs to be provided to you in Python. We exposed the relational operators in 10.0 on geometry objects, we added topological operators in 10.1, and 10.1 SP1 will ship with improved interoperability with WKT and Esri JSON. Another example is with time zones. I like pytz and I'm used to using it, but ArcGIS ships with its own time zone database and its own way of doing things with the time-aware stuff that came with 10.0, so instead of bundling pytz with its own parallel time zone database I went ahead and integrated the ArcGIS way of doing things as arcpy.time, which had the additional benefit of the smart time delta class that uses ArcGIS' time APIs. This then also negated any specific need for dateutil, which is another library I really like. But arcpy.time handles standard Python datetime and tzinfo objects so it all interoperates just fine with everything else, and the skills you learn either from arcpy or pytz/dateutil transfer without a hitch.

Again, I will check back here frequently. Let me know what you think should be considered absolutely essential to Python to include in the ArcGIS stack going forward.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

curtvprice — Wed, 08 Aug 2012 20:21:31 GMT

Thanks Jason. I can see where you could get into a lot of trouble with this, but on the other hand I'm pretty comfortable with suffering with issues with 3rd party libraries that you don't use.

Why Shapely? I've been told that Shapely can do certain things much faster (not having to interact with the ArcGIS system) which is why I included it. This is probably a big reason people are looking to have GDAL available - although the functionality may be there in the Spatial Analyst toolbox, there may be big performance gains using GDAL if integration with the ArcGIS environment (validation, projection, mask etc) is not needed for a geoprocessing use case.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

KimOllivier — Wed, 08 Aug 2012 21:04:11 GMT

Shipping on the CD is not very helpful.

Do you really mean that these modules should be installed with Python?

I have a current problem writing some tools to generate Excel spreadsheets as final reports. Easy to do using the win32com module that comes with win32all or Pythonwin that is included with the standard install but not installed with ArcGIS and Python.

The result is that scripts will work in a development environment but cannot be deployed because the modules are not organisation wide.

Is every IT administration department as difficult to deal with as my students encounter?
They are very unhelpful installing service patches, python modules, customising Window to include file extensions, and dozens of other settings that impact on ArcGIS users. I suggest various system default settings to optimise ArcGIS and they just roll their eyes.

A very serious performance hit is placing the default.gdb in the user's profile area which is not local, but across the network.

So rather than worry about some optional and less used modules, I would rather see the defaults loaded with the software to tackle intransigent and unskilled administrators. I suppose they can always reverse them to 'comply with Policy'.

As to having the lowest common denominator in Linux not having win32 capability? That is not an argument for not having it for the majority that do use Windows applications. After all ArcMap only works on Windows, by that argument we should not have that because it does not run on Linux.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

MathewCoyle — Wed, 08 Aug 2012 21:14:29 GMT

Please keep in mind that we can't include everything requested as there is a lot of work in testing and integration for each library ([Windows|Linux]x[32 bit|64 bit]x[Many other factors]).

Completely understandable, and they don't necessarily have to be default installs. It would be nice to have them as options where it is clear to the user/installer that these are not directly supported, third party modules, and to use at your own risk. It would also be nice to know (I haven't seen any resource for this yet) examples of third party modules and use cases popular internally with Esri, even if they are not standard or supported.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

KimOllivier — Wed, 08 Aug 2012 21:18:31 GMT

Another example is with time zones. I like pytz and I'm used to using it, but ArcGIS ships with its own time zone database and its own way of doing things with the time-aware stuff that came with 10.0, so instead of bundling pytz with its own parallel time zone database I went ahead and integrated the ArcGIS way of doing things as arcpy.time, which had the additional benefit of the smart time delta class that uses ArcGIS' time APIs. This then also negated any specific need for dateutil, which is another library I really like. But arcpy.time handles standard Python datetime and tzinfo objects so it all interoperates just fine with everything else, and the skills you learn either from arcpy or pytz/dateutil transfer without a hitch.

When I backported GPX to Features in 10.1 I found that the time conversion tools do not work even at SP5 so I have to go back to dateutil.

Re: Default geodatabase location

curtvprice — Wed, 08 Aug 2012 21:20:06 GMT

A very serious performance hit is placing the default.gdb in the user's profile area which is not local, but across the network.

Kim, this is a bit off-topic, but there is a fix for this in 10.0 SP5 and 10.1.

HowTo: Set the default Home folder and geodatabase location for new map documents

Of course this requires getting SP5 installed!

In ArcMap 10.1 this setting is exposed in the options button on the Catalog window.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

MathewCoyle — Wed, 08 Aug 2012 21:35:50 GMT

There is a fix for this in 10.0 SP5 and 10.1.

HowTo: Set the default Home folder and geodatabase location for new map documents

Of course this requires getting SP5 installed!

If you are hacking the registry anyways, you can accomplish 90% of that in any version of 10.0. You can set the default geoprocessing output to any gdb location you want.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

JasonScheirer — Wed, 08 Aug 2012 21:43:10 GMT

When I backported GPX to Features in 10.1 I found that the time conversion tools do not work even at SP5 so I have to go back to dateutil.

Did you talk to tech support about it? What tool exactly didn't work?

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

JasonScheirer — Wed, 08 Aug 2012 21:49:41 GMT

Shipping on the CD is not very helpful.

Do you really mean that these modules should be installed with Python?

I have a current problem writing some tools to generate Excel spreadsheets as final reports. Easy to do using the win32com module that comes with win32all or Pythonwin that is included with the standard install but not installed with ArcGIS and Python.

The result is that scripts will work in a development environment but cannot be deployed because the modules are not organisation wide.

...

So rather than worry about some optional and less used modules, I would rather see the defaults loaded with the software to tackle intransigent and unskilled administrators. I suppose they can always reverse them to 'comply with Policy'.

While I sympathize with your situation and I'm not a big fan having my machines locked down by a bunch of (potentially antagonistic) people across campus, I have to say that shipping on the CD but not laying down that extra footprint on disk for the 95% of users who aren't going to use it is the preferred default course of action. It seems what you're asking us to do is modify the installer in a fashion that circumvents/undermines your local IT department, who you do not like. That is not really a use case I can see myself being able to justify to others.

At the UC this year we saw the need for some more documentation on including modules and packages install-free, so we're working on a blog post about that right now.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

curtvprice — Thu, 09 Aug 2012 17:04:32 GMT

At the UC this year we saw the need for some more documentation on including modules and packages install-free, so we're working on a blog post about that right now.

I'm all ears.

There is something to be said for widening the footprint of python modules (within reason of course) enough to provide script developers more functionality with no customization of a standard ArcGIS install. Requiring special futzing can be death to a potentially useful tool!

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

RichardSignell — Thu, 09 Aug 2012 21:01:00 GMT

Even though I voted for some python modules to be included, what I'd really like to see is that ESRI work with Enthought to make sure there is a version of the Enthought Python Distribution that is compatible with each version of ArcGIS. That way, I can just use the 100+ libraries already provided in the EPD together with ArcGIS. This is already true in the latest Enthought EPD 7.3 and ArcGIS 10.1, both of which use Python 2.7 and Numpy 1.6. So everything works very nicely together. But how long will this very nice situation last? Let's make it last longer than just these current releases!

-Rich Signell

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

PhilMorefield — Mon, 13 Aug 2012 13:57:32 GMT

Enthought Python Distribution (EPD)

😉

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

PhilMorefield — Mon, 13 Aug 2012 13:59:14 GMT

Even though I voted for some python modules to be included, what I'd really like to see is that ESRI work with Enthought to make sure there is a version of the Enthought Python Distribution that is compatible with each version of ArcGIS. That way, I can just use the 100+ libraries already provided in the EPD together with ArcGIS. This is already true in the latest Enthought EPD 7.3 and ArcGIS 10.1, both of which use Python 2.7 and Numpy 1.6. So everything works very nicely together. But how long will this very nice situation last? Let's make it last longer than just these current releases!

-Rich Signell

I couldn't agree more. Great idea.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

MarcHoogerwerf — Tue, 14 Aug 2012 18:54:35 GMT

I would like to see requests shipped with ArcGIS. I've been using it to interact with AGOL and I think it makes communication between ArcGIS and AGOL a lot easier with only a few lines of code. Example uses: Change the reference to feature services in all my published webmaps from services running on one AGS machine to services running on another AGS machine.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

KerryAlley — Thu, 11 Oct 2012 13:42:10 GMT

Would there be any interest in Pandas? It would be nice to interact with and manipulate tabular information without actually changing the source table, or to populate tables (before writing them to a geodatabase) without being restricted by InsertRow. Data analysis/processing can be especially nasty in Python/arcpy when creating field values that depend on the values in other rows. I often find myself switching to R (a statistical programming environment... Pandas gives Python an R-like functionality) for the ease of working with tabular data that can be accessed by row/column indices or by row/column names.

Kerry Alley
GIS technician
VT Agency of Transportation

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

VidmasKondratas — Thu, 11 Oct 2012 15:17:13 GMT

The inclusion of numpy and matplotlib was a great addition. It makes using these modules in Esri's GP arcpy environment a lot easier. The addition of Scipy would be fantastic. This module works great with numpy and matplotlib and is used a lot in the academic community. Scipy has a lot of useful matrix operations and would be useful with ArcGIS tools such as Spatial Statistics.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

PhilMorefield — Thu, 11 Oct 2012 18:35:35 GMT

Would there be any interest in Pandas? It would be nice to interact with and manipulate tabular information without actually changing the source table, or to populate tables (before writing them to a geodatabase) without being restricted by InsertRow. Data analysis/processing can be especially nasty in Python/arcpy when creating field values that depend on the values in other rows. I often find myself switching to R (a statistical programming environment... Pandas gives Python an R-like functionality) for the ease of working with tabular data that can be accessed by row/column indices or by row/column names.

Kerry Alley
GIS technician
VT Agency of Transportation

This is a fantastic idea! Pandas would be a great addition.

Re: Should Esri ship third-party modules (like scipy) with ArcGIS?

David_JAnderson — Thu, 11 Oct 2012 21:49:08 GMT

I vote for none. A quick check of the python package index shows almost 25,000 Python modules available. Where would the line be drawn.
What I would like to see is for ESRI to install Python using the standard Python installation directories, then install ArcPy as just another package under the site-packages directory. That way all the off the shelf Python stuff works just fine.

I'm still on 10 so maybe 10.1 fixes this issue.