recommended way of installing geopandas & fiona with arcpy

Hi All,

I installed geopandas after updating to ArcGIS Pro 3.0, and the issue I had above with read_file() has disappeared.

Notably, when I install geopandas, instead of installing fiona 1.8.13, it installs fiona 1.8.21. Also, gdal version in Pro 3.0 is gdal 3.4.0 instead of 3.3.0.

So for what it's worth, updating to Pro 3.0 seems to have fixed the issue for me--though I'm bummed at the prospect that this will likely crop up again when ESRI releases future updates Pro updates.

All comments

geopandas 0.9 ?

fiona 1.8.13.0 ?

glw

yes that is correct.

Is there any follow-up on this topic? I had the same issue with Pro 2.8 and still have it with Pro 2.9.

These are the versions of geopandas and fiona that I have:

And this is what happens when I import fiona or run geopandas.read_file()

As a test, I installed geopandas in a non-arcgis environment and it works fine

In the non-arcgis environment (created by running conda create geopandas-test), these are the versions of fiona/geopandas after running conda install geopandas:

The versions and channels for geopandas and fiona look to be the same in both the arcgis and non-arcgis environments, but gdal is different: In the arcgis cloned env, gdal is version 3.3.0, while in the non-arcgis env it's 3.0.2. Not sure if this is the reason for the error, but worth noting.

With this information, is there any way to get geopandas and fiona to work in an arcgis environment? I actually can use some geopandas functions, but if it needs fiona it won't work (e.g., geopandas.read_file() requires fiona so I cannot use it currently or it gives me an error).

Thank you,

geopandas/ecosystem.md at 08cb5909b88837a2a963f22a81874287ea2ef5ae · geopandas/geopandas (github.com)

gdal 3.3.0

and I am not sure if osgeo is a full install (if you are familiar with it, compare this folder from base to your clone's )

C:\...Your_Install_folder...\bin\Python\envs\arcgispro-py3\Lib\site-packages\osgeo

If it works in the clone, stick with it until python moves beyond 3.7 at least

Hi Dan,

I compared the osgeo folders for the two environments:

For the arcgis-pro based environment (clone of arcgispro-py3; fiona doesn't load on it):

Then for the non-arcgis environment (in which fiona correctly loads):

The most notable difference is that the arcgis env contains a script utils.py,while the non-arcgis version does not contain it.

You will have to examine the __init__ and utils scripts to see what they contain

GARYLESTER

I have had this issue before, I think you should get the precompiled whl files from here https://www.lfd.uci.edu/~gohlke/pythonlibs/ . I downloaded gdal,fiona and geopandas. First I uninstalled the gdal version that comes with arcgisPro then I installed gdal,fiona then geopandas. I have only had luck installing these through command line. Make sure you download the precompiled library for your specific python version and the versions of fiona and gdal must be compatible.

Hi Gary,

Thanks for the tip. I tried the following:

create new environment as clone of default Pro environment
Ran conda remove gdal --force-remove to remove gdal without also removing arcpy
Ran conda install gdal=3.0.2 -c conda-forge, since gdal 3.0.2 did not have the fiona issue

However, running step 3 said there were various conflicts between gdal 3.0.2 and the other packages in the Pro environment. Specific conflicts are listed below, but critically, it looks like arcpy and arcgis have inconsistencies, and since my goal is to have arcpy and fiona work in the same env, I cannot get rid of arcpy.

The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - esri/win-64::arcgis==1.9.1=py37_2327
  - esri/win-64::arcpy==2.9=py37_arcgispro_32704
  - defaults/win-64::argon2-cffi==20.1.0=py36he774522_1
  - defaults/win-64::ipykernel==5.3.4=py36h5ca1d4c_0
  - defaults/win-64::ipywidgets==7.4.2=py37_0
  - esri/noarch::jupyterlab==2.2.7=py_0
  - defaults/noarch::jupyterlab_server==1.2.0=py_0
  - esri/noarch::jupyter_console==6.4.0=py_1
  - esri/noarch::jupyter_contrib_core==0.3.3=py_4
  - esri/noarch::jupyter_contrib_nbextensions==0.5.1=py_23
  - esri/noarch::jupyter_highlight_selected_word==0.2.0=py_4
  - esri/noarch::jupyter_latex_envs==1.4.6=py_3
  - esri/noarch::jupyter_nbextensions_configurator==0.4.1=py_3
  - defaults/win-64::notebook==6.1.1=py36_0
  - esri/win-64::pro_notebook_integration==2.9=py37_9
  - defaults/win-64::widgetsnbextension==3.4.2=py37_0

Is there any reason why the pre-compiled WHL files are different from conda forge? And did you run into the same conflicts?

Thanks,

Thanks

arcpy and gdal and numpy are tied together for a variety of functions and functionality.

You would be ill advised to mess with the linkage

I have sometimes found I have better luck installing components at the same time in a new environment.

conda create -n geopandas arcpy geopandas fiona

I tried that this morning and it failed me. Import did this to me,

File "C:\Users\bwilson\AppData\Local\ESRI\conda\envs\geopandas\lib\site-packages\rtree\finder.py", line 67, in load
raise OSError("could not find or load {}".format(lib_name))
OSError: could not find or load spatialindex_c-64.dll

Leaving out arcpy gives me a DLL load error in pyproj instead. If I had time I would next try forcing it to use older versions of geopandas.

import pyproj
File "C:\Users\bwilson\AppData\Local\ESRI\conda\envs\geopandas\lib\site-packages\pyproj\__init__.py", line 49, in <module>
import pyproj.network
File "C:\Users\bwilson\AppData\Local\ESRI\conda\envs\geopandas\lib\site-packages\pyproj\network.py", line 10, in <module>
from pyproj._network import ( # noqa: F401 pylint: disable=unused-import
ImportError: DLL load failed while importing _network: The specified module could not be found.

GARYLESTER

Here are the specific versions that worked for me but my argis pro is still python 3.6.

Fiona 1.8.19

GDAL 3.2.3

I am doing all the install and uninstall with pip and .whl files in the command line. I have successfully installed this way on local machine and EC2 machine as well but it is tricky. The python version off the .whl need to match your python version. Other than version between GDAL and the Fiona I have not had any issues installing like this.

Dang. This is a thorny issue. As much as I prefer using geopandas, I've managed to make things work okay with ESRI's Spatially Enabled Data Frames, so for now will use them instead of geopandas, at least for loading data. Hopefully there will eventually be an update of geopandas that can use the same GDAL version that arcpy does.

I tried loading arcgis + geopandas + fiona on Debian 11 and it's broken there too. arcgis and gdal are using incompatible libpoppler libraries, so they won't work at the same time.

It's frustrating when you have to give up significant features and bug fixes in the open source libraries to make the Esri code work.

I've gotten to the point where I keep separate conda environments for each project, and in some few cases I keep multiple environments for one project to work around some of these problems.

That could be the new Esri slogan! "It works okay."

from BUG-000128147: The GeoPandas module fails to install from the ArcGI.. (esri.com)

conda install geopandas libtiff=4.0.10

referencing

ArcGIS Notebooks in ArcGIS Pro (2.6) - Esri Community

and full circle.... it was supposed to be fixed in 2.8

Thanks for following up Dan.

However I just tried the following:

Make clone of default env
Run conda install geopandas libtiff=4.0.10

After running, the terminal spent over a half hour trying to do the installation, throwing up *lots* of warning messages about package conflicts with the versions installed in the default environment, then ultimately ended up not succeeding with the installation.

FYI, I'm using Pro 2.9

it may be time to give Tech Support a call. Sorting out the package issues may not be as simple as it seems.

In the interim, separate your geopandas stuff completely from anything that uses the arcpy environment since open source stuff (gdal, osgeo etc) originates from esri channels for their purposes

sven_aurea

We used to work with ArcGIS pro 2.5.2 I decided to upgrade to ArcGIS pro 2.6 when it released. Well worst choice ever. I just could not get the environment to work anymore and it seems to be the same issue with Fiona as mentioned in this topic. Today after a long time I tried to create the environment with ArcGIS pro 2.9 still the same issues. Crazy this is still a thing after years. Not happy. 😞

Esri is big and slow moving, so even assuming they WANT to support open source projects, they will always be several steps behind. You just have to accept that, and understand that calling support won't change it.

The good part is that with tools like conda and docker available now, there are many ways to address the problem.

feralcatcolonist_old

Pre ArcGIS Pro 3.0; this is the recipe that I used to create a standalone environment with ArcPy and Geopandas within the anaconda command prompt

conda create -n esri python=3.7.11
conda activate esri
conda install -c esri arcpy
conda install -c conda-forge geopandas 
conda install -c conda-forge rtree=0.9.3

Beginning in 3.0 this is what I used; there's an issue with Geopandas so you have to update pyproj afterwards. With 3.0, ArcPy won't import (for me) in a jupyter notebook within VSCode which is a total bummer.

conda create -n esri
conda activate esri
conda install -c esri arcpy
conda install -c conda-forge geopandas 
conda update pyproj