Select to view content in your preferred language

Need to rebuild conda environment after ArcGIS Pro updates

6181
8
01-08-2021 09:59 AM
DarrenConly
Regular Contributor

Hi All,

I recently updated from Pro 2.6 to Pro 2.7 and encountered an issue that I've had in the past when updating Pro. Specifically,

  1. I have a nice conda env set up (i.e. time went in to downloading lots of custom packages) and I use Spyder as my IDE.
  2. But after updating Pro, when I import arcpy in Spyder, I get this error message:

 

The version of the binaries in the active Python environment do
  not match the version of the binaries of the ArcGIS Pro application.

Active environment version : 2.6
arcgispro-py3 environment version : 2.7

To create a new environment that is up to date with arcgispro-py3:
 1. Generate a list of additional packages installed in your current environment,
    with the conda command:
        > conda env export > env.yaml
 2. (Optional) If you have additional dependencies installed through pip,
    find those with pip freeze:
        > pip freeze > requirements.txt
 2. Create a new environment by cloning arcgispro-py3, and activate it:
        > conda create --clone arcgispro-py3 --name my-env --pinned
        > activate my-env
 3. Add back missing conda packages from your current environment:
        > conda env update -n my-env -f env.yaml
 4. (Optional) Add back missing packages from pip:
        > pip install -r requirements.txt
 5. (Optional) Make it the default for the ArcGIS Pro application and the
     "Python Command Prompt":
        > proswap my-env

Traceback (most recent call last):

  File "<ipython-input-1-5467a3dc9fe3>", line 1, in <module>
    import arcpy

  File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\__init__.py", line 74, in <module>
    from arcpy.geoprocessing import gp

  File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\geoprocessing\__init__.py", line 14, in <module>
    from ._base import *

  File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\geoprocessing\_base.py", line 14, in <module>
    import arcgisscripting

  File "C:\Users\dconly\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgisscripting\__init__.py", line 112, in <module>
    raise ImportError("version mis-match") from None

ImportError: version mis-match

 

 

Referring to the excerpt above, I go through the first 3 steps (unfortunately step 2 is labeled twice but I do steps 1 and both of the 2s). But when I try to do the 4th step (run conda env update -n my-env -f env.yaml) I get this message below:

 

UnsatisfiableError: The following specifications were found to be in conflict:
  - matplotlib-base 3.2.2 py36h64f37c6_0 -> python >=3.6,<3.7.0a0
  - python 3.7.*
Use "conda info <package>" to see the dependencies for each package.

 

 

This is the part cannot get past. It seems like some of the packages from my old custom env (as indicated in the yaml) are not compatible with the new version of python in the new custom env.

My other alternative is to just manually re-add all of the packages that were in my old custom environment, but this is tedious (about a dozen packages to add and I need to remember which ones) and should not be as annoying for a process that apparently needs to happen semi-regularly (i.e., when ESRI pushes out an update to Pro that makes it no longer compatible with older versions of arcpy).

Has anyone else seen this? Or found a more efficient way of recreating a custom environment after Pro updates?

8 Replies
DanPatterson
MVP Esteemed Contributor

Dependencies in Spyder 4.2.   Install matplotlib 3.2.1 I suspect the 3.2.2 was just the most recent update.

Spyder's list below works perfectly with Pro 2.7.  There are a couple of other dependencies in the list below that are version specific that Pro had issues with, so make sure you have my versions (eg.  jedi and parso ... long stories behind those)

That may not solve your problem, but it will ensure you have no issues with spyder for sure.

# Mandatory:
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  3.0.4 (OK)
cloudpickle >=0.5.0             :  1.6.0 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree >=3.0.2            :  3.1.0 (OK)
IPython >=4.0                   :  7.18.1 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
nbconvert >=4.0                 :  5.6.1 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
paramiko >=2.4.0                :  2.7.2 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.7.2 (OK)
pygments >=2.0                  :  2.7.0 (OK)
pylint >=1.0                    :  2.6.0 (OK)
pyls >=0.36.1;<1.0.0            :  0.36.2 (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.1.1             :  0.3.0 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.1 (OK)
qtconsole >=4.7.7               :  4.7.7 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
rtree >=0.8.3                   :  0.9.4 (OK)
setuptools >=39.0.0             :  51.0.0.post20201207 (OK)
sphinx >=0.6.6                  :  3.2.1 (OK)
spyder_kernels >=1.10.0;<1.11.0 :  1.10.0 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog >=0.10.3               :  0.10.4 (OK)
zmq >=17                        :  19.0.2 (OK)

# Optional:
cython >=0.21                   :  None (NOK)
matplotlib >=2.0.0              :  3.3.1 (OK)
numpy >=1.7                     :  1.19.2 (OK)
pandas >=1.1.1                  :  1.1.5 (OK)
scipy >=0.17.0                  :  1.5.2 (OK)
sympy >=0.7.3                   :  1.5.1 (OK)

... sort of retired...
DarrenConly
Regular Contributor

Thanks @DanPatterson. I got Spyder to work. And I can manually reinstall the packages that I used in my old package, but I wish there was a way I could just port over the non-standard packages from my old env to the new one as a bulk process rather than reinstalling them manually one by one. In theory this is doable with a .yml of the old conda environment, but when I try to load it I get the version conflicts like with matplotlib.

Let's say you have 20 packages in a custom environment, then you apply a Pro update that requires you to restart from the default environment. How would you make sure those 20 packages get transferred to the new environment? Would you install them one by one? Or is there a more efficent/bulk method of doing so?

0 Kudos
DanPatterson
MVP Esteemed Contributor

@DarrenConly I normally don't have that many to bring over so I just copy and paste the command line from a text file and enjoy a coffee.

I don't use clones or separate environments either, I install everything in the base environment and keep track of dependencies as needed.

pip throws another lurch into things since Pro isn't using current conda which supposedly handles things better.

If things go bad .... > conda install --revision 1 

is my go to.

Conda... the dependency trail - GeoNet, The Esri Community

 


... sort of retired...
0 Kudos
DarrenConly
Regular Contributor

Thanks Dan,

I've considered just updating the base env (arcgispro-py3) before. Do you know if doing so might fix the issue? E.g., if I add a custom package to the base environment, then Pro does an update, will that reset the base environment?

At this point I've just manually re-added the packages (I also keep a list of custom packages), but I feel like there should be a more efficient way of doing it than doing it one-by-one

0 Kudos
MarkGilbert
Esri Contributor

Conda has a few commands to help with backing up and sharing environments. I wonder if there is a way for you to use these tools to quickly get back all your custom packages installs after a Pro update.

Check this out Sharing an Environment in the docs.

0 Kudos
DarrenConly
Regular Contributor

Hi @MarkGilbert ,

Conda does have the option to save an environment as a YML file then to import it into a new environment as an update to the environment, but the problem for me is that some of the versions in the old environment were not compatible in the new environment (see the excerpt showing Matplotlib version conflict in my original post), so I couldn't just bulk move custom packages over and instead needed to reinstall them individually.

0 Kudos
MarkGilbert
Esri Contributor

Ah, I see. Didn't catch that part. I'm constantly struggling with Pro upgrades and how to get my preferred custom env updated with the latest arcpy components and not lose my other packages. I've just been keeping a list of packages and adding them back in after creating a new cloned environment. 

PaulCyr1
Occasional Contributor

This happens to me every time too so I don't do ArcGIS PRO updates unless they have a good reason as it take a week to get my DEV environment back to a usable state. It is a total pain in the you know where, if ESRI could just add pyodbc & spyder as approved addin modules for me that would make my life much easier!