Should Esri ship third-party modules (like scipy) with ArcGIS?

[curtvprice]

The ArcGIS Python 2.7 distribution for 10.1 includes (in addition to the python standard library)  the user-contributed modules numpy and matplotlib. It would be really great to include some more modules. If you agree, vote this item up and add your favorites (and use cases) in the comments.

Vote here, and also feel free to vote the idea up:

ideas.esri.com: Ship additional python extensions

[JasonScheirer]

Shipping on the CD is not very helpful.

Do you really mean that these modules should be installed with Python?

I have a current problem writing some tools to generate Excel spreadsheets as final reports. Easy to do using the win32com module that comes with win32all or Pythonwin that is included with the standard install but not installed with ArcGIS and Python.

The result is that scripts will work in a development environment but cannot be deployed because the modules are not organisation wide.

...

So rather than worry about some optional and less used modules, I would rather see the defaults loaded with the software to tackle intransigent and unskilled administrators. I suppose they can always reverse them to 'comply with Policy'.

While I sympathize with your situation and I'm not a big fan having my machines locked down by a bunch of (potentially antagonistic) people across campus, I have to say that shipping on the CD but not laying down that extra footprint on disk for the 95% of users who aren't going to use it is the preferred default course of action. It seems what you're asking us to do is modify the installer in a fashion that circumvents/undermines your local IT department, who you do not like. That is not really a use case I can see myself being able to justify to others.

At the UC this year we saw the need for some more documentation on including modules and packages install-free, so we're working on a blog post about that right now.

[curtvprice]

At the UC this year we saw the need for some more documentation on including modules and packages install-free, so we're working on a blog post about that right now.

I'm all ears.

There is something to be said for widening the footprint of python modules (within reason of course) enough to provide script developers more functionality with no customization of a standard ArcGIS install. Requiring special futzing can be death to a potentially useful tool!

[RichardSignell]

Even though I voted for some python modules to be included, what I'd really like to see is that ESRI work with Enthought to make sure there is a version of the Enthought Python Distribution that is compatible with each version of ArcGIS.  That way, I can just use the 100+ libraries already provided in the EPD together with ArcGIS.   This is already true in the latest Enthought EPD 7.3 and ArcGIS 10.1, both of which use Python 2.7 and Numpy 1.6.   So everything works very nicely together.  But how long will this very nice situation last?    Let's make it last longer than just these current releases!

-Rich Signell

[PhilMorefield]

Another commenter suggested ESRI work with Enthought, Inc. to coordinate on an ArcGIS-compatible version of the Enthought Python Distribution (EPD) at every release of ArcGIS. This is a great idea, and would render polls like this unnecessary! 😉

I think there are a lot of questions on the forums that could be preemptively answered with code analysis tools, like pylint. How about including a free, lightweight Python IDE that already has pylint integrated? I know the ESRI folks are fans of Pyscripter, which is great. Spyder is another possiblity. IEP is another solid IDE, though it lacks pylint integration at present.

I don't see much value in only bundling packages like gdal and shapely. 99% of the functionality in those modules is already included in ArcGIS. PySAL has some nice tools but, again, is already included in the EPD.

As an aside, I see netCDF4 listed above. I've found this to be incompatible with ArcGIS 9.x and 10.0 sp5 on Windows XP 32-bit.

Great thread!

[PhilMorefield]

Even though I voted for some python modules to be included, what I'd really like to see is that ESRI work with Enthought to make sure there is a version of the Enthought Python Distribution that is compatible with each version of ArcGIS.  That way, I can just use the 100+ libraries already provided in the EPD together with ArcGIS.   This is already true in the latest Enthought EPD 7.3 and ArcGIS 10.1, both of which use Python 2.7 and Numpy 1.6.   So everything works very nicely together.  But how long will this very nice situation last?    Let's make it last longer than just these current releases!

-Rich Signell

I couldn't agree more. Great idea.

[MarcHoogerwerf]

I would like to see requests shipped with ArcGIS. I've been using it to interact with AGOL and I think it makes communication between ArcGIS and AGOL a lot easier with only a few lines of code. Example uses: Change the reference to feature services in all my published webmaps from services running on one AGS machine to services running on another AGS machine.

[KerryAlley]

Would there be any interest in Pandas?  It would be nice to interact with and manipulate tabular information without actually changing the source table, or to populate tables (before writing them to a geodatabase) without being restricted by InsertRow.  Data analysis/processing can be especially nasty in Python/arcpy when creating field values that depend on the values in other rows.  I often find myself switching to R (a statistical programming environment... Pandas gives Python an R-like functionality) for the ease of working with tabular data that can be accessed by row/column indices or by row/column names.

Kerry Alley
GIS technician
VT Agency of Transportation

[VidmasKondratas]

The inclusion of numpy and matplotlib was a great addition. It makes using these modules in Esri's GP arcpy environment a lot easier. The addition of Scipy would be fantastic. This module works great with numpy and matplotlib and is used a lot in the academic community. Scipy has a lot of useful matrix operations and would be useful with ArcGIS tools such as Spatial Statistics.

[PhilMorefield]

Would there be any interest in Pandas?  It would be nice to interact with and manipulate tabular information without actually changing the source table, or to populate tables (before writing them to a geodatabase) without being restricted by InsertRow.  Data analysis/processing can be especially nasty in Python/arcpy when creating field values that depend on the values in other rows.  I often find myself switching to R (a statistical programming environment... Pandas gives Python an R-like functionality) for the ease of working with tabular data that can be accessed by row/column indices or by row/column names.

Kerry Alley
GIS technician
VT Agency of Transportation

This is a fantastic idea! Pandas would be a great addition.

[David_JAnderson]

I vote for none.  A quick check of the python package index shows almost 25,000 Python modules available.  Where would the line be drawn.
What I would like to see is for ESRI to install Python using the standard Python installation directories, then install ArcPy as just another package under the site-packages directory.  That way all the off the shelf Python stuff works just fine.

I'm still on 10 so maybe 10.1 fixes this issue.