nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Should Esri ship third-party modules (like scipy) with ArcGIS?

curtvprice

The ArcGIS Python 2.7 distribution for 10.1 includes (in addition to the python standard library) the user-contributed modules numpy and matplotlib. It would be really great to include some more modules. If you agree, vote this item up and add your favorites (and use cases) in the comments.

Vote here, and also feel free to vote the idea up:

ideas.esri.com: Ship additional python extensions

Find more posts tagged with

Accepted answers

All comments

MathewCoyle

pywin32 specifically the win32api.

I like to use it for quasi OS authentication by accessing active directory user names and permissions. As a custom install right now it is a hassle to package it with the standard deployment.

JasonScheirer

I will be visiting this thread fairly often to keep up-to-date on what you all think. We on the Python team want you to be happy and producting using Python on the ArcGIS stack.

Please keep in mind that we can't include everything requested as there is a lot of work in testing and integration for each library ([Windows|Linux]x[32 bit|64 bit]x[Many other factors]). For example, I had to help with a workaround for a memory leak in matplotlib that was causing many spatial statistics tools to leak huge amounts of memory. And what would pywin32 look like on Linux? Adding a third-party module means adding an additional dependency in ArcGIS on that third-party which may or may not update regularly or anywhere in sync with our own timetables. Our release dates and the release dates of the third party can result in scares for us: numpy didn't have support for Python 2.7 until a week before 10.1 beta's cutoff date, so we almost had to stick with 2.6 for 10.1. I can only imagine how much closer (or over?) we would have run if we had a scipy dependency on top of that. And the opposite is true, too: we started with upgrading to 2.7 in our daily builds when it was first released, updated and tested 10.1 to Python 2.7.2 during the development cycle and then 2.7.3 came out right before 10.1's release date. I can see one or two libraries being frozen at a version you don't like (maybe they're crashy or have a bug that only existed for a few builds or have awkward APIs which are later fixed) on every install of ArcGIS, and fear that could cause more pain than the hassle of shipping your own with your tools. Not to mention the entirely unknown quantity of not just porting ourselves to Python 3, but waiting for all our third-party dependencies to port as well.

There's also reduplication of functionality in a lot of these libraries. I agree that we should get our GDAL bindings actually shipping and not just on the resource center. But on the other side, there's no realistic use case for Shapely considering ArcGIS already handles geometry in a very flexible, intelligent way. The only missing part is that it needs to be provided to you in Python. We exposed the relational operators in 10.0 on geometry objects, we added topological operators in 10.1, and 10.1 SP1 will ship with improved interoperability with WKT and Esri JSON. Another example is with time zones. I like pytz and I'm used to using it, but ArcGIS ships with its own time zone database and its own way of doing things with the time-aware stuff that came with 10.0, so instead of bundling pytz with its own parallel time zone database I went ahead and integrated the ArcGIS way of doing things as arcpy.time, which had the additional benefit of the smart time delta class that uses ArcGIS' time APIs. This then also negated any specific need for dateutil, which is another library I really like. But arcpy.time handles standard Python datetime and tzinfo objects so it all interoperates just fine with everything else, and the skills you learn either from arcpy or pytz/dateutil transfer without a hitch.

Again, I will check back here frequently. Let me know what you think should be considered absolutely essential to Python to include in the ArcGIS stack going forward.

curtvprice

Thanks Jason. I can see where you could get into a lot of trouble with this, but on the other hand I'm pretty comfortable with suffering with issues with 3rd party libraries that you don't use.

Why Shapely? I've been told that Shapely can do certain things much faster (not having to interact with the ArcGIS system) which is why I included it. This is probably a big reason people are looking to have GDAL available - although the functionality may be there in the Spatial Analyst toolbox, there may be big performance gains using GDAL if integration with the ArcGIS environment (validation, projection, mask etc) is not needed for a geoprocessing use case.

KimOllivier

Shipping on the CD is not very helpful.

Do you really mean that these modules should be installed with Python?

I have a current problem writing some tools to generate Excel spreadsheets as final reports. Easy to do using the win32com module that comes with win32all or Pythonwin that is included with the standard install but not installed with ArcGIS and Python.

The result is that scripts will work in a development environment but cannot be deployed because the modules are not organisation wide.

Is every IT administration department as difficult to deal with as my students encounter?
They are very unhelpful installing service patches, python modules, customising Window to include file extensions, and dozens of other settings that impact on ArcGIS users. I suggest various system default settings to optimise ArcGIS and they just roll their eyes.

A very serious performance hit is placing the default.gdb in the user's profile area which is not local, but across the network.

So rather than worry about some optional and less used modules, I would rather see the defaults loaded with the software to tackle intransigent and unskilled administrators. I suppose they can always reverse them to 'comply with Policy'.

As to having the lowest common denominator in Linux not having win32 capability? That is not an argument for not having it for the majority that do use Windows applications. After all ArcMap only works on Windows, by that argument we should not have that because it does not run on Linux.

MathewCoyle

Please keep in mind that we can't include everything requested as there is a lot of work in testing and integration for each library ([Windows|Linux]x[32 bit|64 bit]x[Many other factors]).

Completely understandable, and they don't necessarily have to be default installs. It would be nice to have them as options where it is clear to the user/installer that these are not directly supported, third party modules, and to use at your own risk. It would also be nice to know (I haven't seen any resource for this yet) examples of third party modules and use cases popular internally with Esri, even if they are not standard or supported.

KimOllivier

Another example is with time zones. I like pytz and I'm used to using it, but ArcGIS ships with its own time zone database and its own way of doing things with the time-aware stuff that came with 10.0, so instead of bundling pytz with its own parallel time zone database I went ahead and integrated the ArcGIS way of doing things as arcpy.time, which had the additional benefit of the smart time delta class that uses ArcGIS' time APIs. This then also negated any specific need for dateutil, which is another library I really like. But arcpy.time handles standard Python datetime and tzinfo objects so it all interoperates just fine with everything else, and the skills you learn either from arcpy or pytz/dateutil transfer without a hitch.

When I backported GPX to Features in 10.1 I found that the time conversion tools do not work even at SP5 so I have to go back to dateutil.

gpx_v10.zip

curtvprice

A very serious performance hit is placing the default.gdb in the user's profile area which is not local, but across the network.

Kim, this is a bit off-topic, but there is a fix for this in 10.0 SP5 and 10.1.

HowTo: Set the default Home folder and geodatabase location for new map documents

Of course this requires getting SP5 installed!

In ArcMap 10.1 this setting is exposed in the options button on the Catalog window.

MathewCoyle

There is a fix for this in 10.0 SP5 and 10.1.

HowTo: Set the default Home folder and geodatabase location for new map documents

Of course this requires getting SP5 installed!

If you are hacking the registry anyways, you can accomplish 90% of that in any version of 10.0. You can set the default geoprocessing output to any gdb location you want.

JasonScheirer

When I backported GPX to Features in 10.1 I found that the time conversion tools do not work even at SP5 so I have to go back to dateutil.

Did you talk to tech support about it? What tool exactly didn't work?

JasonScheirer

Shipping on the CD is not very helpful.

Do you really mean that these modules should be installed with Python?

I have a current problem writing some tools to generate Excel spreadsheets as final reports. Easy to do using the win32com module that comes with win32all or Pythonwin that is included with the standard install but not installed with ArcGIS and Python.

The result is that scripts will work in a development environment but cannot be deployed because the modules are not organisation wide.

...

So rather than worry about some optional and less used modules, I would rather see the defaults loaded with the software to tackle intransigent and unskilled administrators. I suppose they can always reverse them to 'comply with Policy'.

While I sympathize with your situation and I'm not a big fan having my machines locked down by a bunch of (potentially antagonistic) people across campus, I have to say that shipping on the CD but not laying down that extra footprint on disk for the 95% of users who aren't going to use it is the preferred default course of action. It seems what you're asking us to do is modify the installer in a fashion that circumvents/undermines your local IT department, who you do not like. That is not really a use case I can see myself being able to justify to others.

At the UC this year we saw the need for some more documentation on including modules and packages install-free, so we're working on a blog post about that right now.

curtvprice

At the UC this year we saw the need for some more documentation on including modules and packages install-free, so we're working on a blog post about that right now.

I'm all ears.

There is something to be said for widening the footprint of python modules (within reason of course) enough to provide script developers more functionality with no customization of a standard ArcGIS install. Requiring special futzing can be death to a potentially useful tool!

rsignell

Even though I voted for some python modules to be included, what I'd really like to see is that ESRI work with Enthought to make sure there is a version of the Enthought Python Distribution that is compatible with each version of ArcGIS. That way, I can just use the 100+ libraries already provided in the EPD together with ArcGIS. This is already true in the latest Enthought EPD 7.3 and ArcGIS 10.1, both of which use Python 2.7 and Numpy 1.6. So everything works very nicely together. But how long will this very nice situation last? Let's make it last longer than just these current releases!

-Rich Signell

KaraUtter_1-1652826474532.png

PhilMorefield

Another commenter suggested ESRI work with Enthought, Inc. to coordinate on an ArcGIS-compatible version of the Enthought Python Distribution (EPD) at every release of ArcGIS. This is a great idea, and would render polls like this unnecessary! 😉

I think there are a lot of questions on the forums that could be preemptively answered with code analysis tools, like pylint. How about including a free, lightweight Python IDE that already has pylint integrated? I know the ESRI folks are fans of Pyscripter, which is great. Spyder is another possiblity. IEP is another solid IDE, though it lacks pylint integration at present.

I don't see much value in only bundling packages like gdal and shapely. 99% of the functionality in those modules is already included in ArcGIS. PySAL has some nice tools but, again, is already included in the EPD.

As an aside, I see netCDF4 listed above. I've found this to be incompatible with ArcGIS 9.x and 10.0 sp5 on Windows XP 32-bit.

Great thread!

PhilMorefield

Even though I voted for some python modules to be included, what I'd really like to see is that ESRI work with Enthought to make sure there is a version of the Enthought Python Distribution that is compatible with each version of ArcGIS. That way, I can just use the 100+ libraries already provided in the EPD together with ArcGIS. This is already true in the latest Enthought EPD 7.3 and ArcGIS 10.1, both of which use Python 2.7 and Numpy 1.6. So everything works very nicely together. But how long will this very nice situation last? Let's make it last longer than just these current releases!

-Rich Signell

I couldn't agree more. Great idea.

MarcHoogerwerf_TAP

I would like to see requests shipped with ArcGIS. I've been using it to interact with AGOL and I think it makes communication between ArcGIS and AGOL a lot easier with only a few lines of code. Example uses: Change the reference to feature services in all my published webmaps from services running on one AGS machine to services running on another AGS machine.

KerryAlley

Would there be any interest in Pandas? It would be nice to interact with and manipulate tabular information without actually changing the source table, or to populate tables (before writing them to a geodatabase) without being restricted by InsertRow. Data analysis/processing can be especially nasty in Python/arcpy when creating field values that depend on the values in other rows. I often find myself switching to R (a statistical programming environment... Pandas gives Python an R-like functionality) for the ease of working with tabular data that can be accessed by row/column indices or by row/column names.

Kerry Alley
GIS technician
VT Agency of Transportation

VidmasKondratas

The inclusion of numpy and matplotlib was a great addition. It makes using these modules in Esri's GP arcpy environment a lot easier. The addition of Scipy would be fantastic. This module works great with numpy and matplotlib and is used a lot in the academic community. Scipy has a lot of useful matrix operations and would be useful with ArcGIS tools such as Spatial Statistics.

PhilMorefield

Would there be any interest in Pandas? It would be nice to interact with and manipulate tabular information without actually changing the source table, or to populate tables (before writing them to a geodatabase) without being restricted by InsertRow. Data analysis/processing can be especially nasty in Python/arcpy when creating field values that depend on the values in other rows. I often find myself switching to R (a statistical programming environment... Pandas gives Python an R-like functionality) for the ease of working with tabular data that can be accessed by row/column indices or by row/column names.

Kerry Alley
GIS technician
VT Agency of Transportation

This is a fantastic idea! Pandas would be a great addition.

David_JAnderson

I vote for none. A quick check of the python package index shows almost 25,000 Python modules available. Where would the line be drawn.
What I would like to see is for ESRI to install Python using the standard Python installation directories, then install ArcPy as just another package under the site-packages directory. That way all the off the shelf Python stuff works just fine.

I'm still on 10 so maybe 10.1 fixes this issue.

PhilMorefield

I vote for none. A quick check of the python package index shows almost 25,000 Python modules available. Where would the line be drawn.
What I would like to see is for ESRI to install Python using the standard Python installation directories, then install ArcPy as just another package under the site-packages directory. That way all the off the shelf Python stuff works just fine.

I'm still on 10 so maybe 10.1 fixes this issue.

Well I think the line would be drawn based on feedback received in threads like this one. Python is becoming the standard for scientific software, and I'm sure that out of 25,000 modules a handfull are broad enough so as to be almost universally useful to the GIS user community.

I agree that the non-standard Python installation ESRI uses is annoying. But this is standard practice for other scientific/GIS applications that need a Python installation (e.g., Quantum GIS does the same thing). The workaround to this is just install Python, numpy, matplotlib, etc. first so the ArcGIS installer will use those installations instead of putting in its own. The arcpy software is still not put into the 'site-packages' folder like other modules, but at least your Python installation will look normal in every other way.

Celeste_SuliinBurris

I would like to see pandas. Apparently the pre-compiled version of pandas from <http://www.lfd.uci.edu/~gohlke/pythonlibs/#pandas> won't work with the version of numpy that ships with ArcGIS 10.1. I have installed it carefully, but I keep getting "numpy.dtype has the wrong size, try recompiling"

MikeMacRae

+1 for win32com. I do a lot of reporting based on spatial data and make use of a lot of excel spreadsheets. Whatever I can't get ArcGIS to do to communicate with excel, I can do through this module.

JasonScheirer

Mike: we're bundling xlrd and xlwt with newer versions of ArcGIS, would that be adequate for your needs?

I'm aware of the demand for Pandas (I want it too), I'm going to see what I can do with that.

RobertBorchert

Kind of against ESRI shipping any 3rd party items from a liability point of view. If it is not something they provide support for perhaps they should provide a link for you to go get it or provide some sort of vehical.

MikeMacRae

Mike: we're bundling xlrd and xlwt with newer versions of ArcGIS, would that be adequate for your needs?

I'm aware of the demand for Pandas (I want it too), I'm going to see what I can do with that.

Hey Jason, when I first started learning how to manipulate spreasheets with python, the first modules I tried were the python-excel modules. They are very easy to learn. The last time I used python-excel was a while ago, though, and I remember it being very basic and lacked the ability to do some more advanced formatting or formula building. Maybe there have been updates since I've lasted used it though.

I then went on to win32com because it uses com objects which I believe can be applied to any of the MS Office products (not that I've used it for anything outside of excel yet) but if that's the case, I would prefer win32com. I use it quite extensively with reports and the more I learn how to manipulate spreadsheets, the more I like it. It's nice to get used to how a module works and then be able to apply the same logic with other applications, instead of hacking amongst modules trying to finding tools that work together.

curtvprice

Pandas sounds like a really good idea.

curtvprice

Talking with the arcgis geoprocessing team at UC yesterday -- they said that ArcGIS Pro beta 4 (python 3.4 of course) is shipped with an "internal" python install used by the app and a separate installer with python 3.4 that includes what Bill M was calling the "sci stack" which includes the modules used by esri (numpy, matplotlib etc) plus a another 300MB of so of other scientific modules (I think he said scipy, pandas, netCDF4) that aren't used by Esri but are useful to us.

The idea is they don't have the footprint there in a standard Pro install, but the group of users that will be installing this separate python installer are developers and can deal with the footprint and any issues they run into with these modules and not blame Esri. 🙂

This is great news. Hope there is a blog post coming from the arcpy crew soon! Maybe on the cafe?

ChristopherMoravec1

Another option that would be great is for Esri to simply include pip and Easy Install. This would make it easy for people to write scripts in Python or even BAT files that could add any 3rd party libraries needed. Most Python distributions (excluding windows) come with pip and Easy Install ready to go, it seems like something Esri could add that would make it more "pythonic" and much easier for anyone to require additional libraries or dependencies.

MatejTacer

I would support adding pip and/or easy_install rather than including too many third party libraries.

Using pip or easy_install is super easy and quick to install whatever library you might need.

JasonScheirer

pip is bundled with Python 3.4.

curtvprice

ArcGIS 10.3.1 and ArcGIS Pro 1.0 include optional installers for Python that include a rich standard ArcGIS Python stack. SciPy and pandas are included.

ArcGIS Blog: Strengthening the Link between GIS and Science

WesMiller

I would like to see a module capable of creating a form

RussellRoberts1_0-1652879485480.png

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.