ArcGIS Pro 2.8 Python Environment not seeing package changes (In-app)

[John_Spence]

Is anyone else seeing some odd behaviors with the custom python environments?  I did a full uninstall and started over from scratch just in case.

Issue:

When trying to install a package from within ArcGIS Pro into a custom environment, the package appears to not install.  If you do it via command (ie. python.exe -m pip install XXXXX), it works like a champ, but the software does not see the package as having been installed.

If using -m pip install on the command line, the package shows up, is accessible, etc.  Typical behaviors one would expected there.

Expected behavior:

Python packages install as they did for 2.7.3 and the interface reflects the change.

[DanPatterson]

You are switching into the clone environment through Pro I assume (?)

What does "conda" show for that environment?

ie...

(arcgispro-py3) C:\YourInstallFolder\bin\Python\envs\arcgispro-py3>conda list
# packages in environment at C:\YourInstallFolder\bin\Python\envs\arcgispro-py3:
#
# Name Version Build Channel
alabaster 0.7.12 py37_0
appdirs 1.4.4 py_0
arcgis 1.8.5 py37_1783 esri
arcgispro 2.8 0 esri
arcpy 2.8 py37_arcgispro_29734 [arcgispro] esri

.... snip ....

and for the clone ....

(arcgispro-py3) C:\YourInstallFolder\bin\Python\envs\arcgispro-yourclone>conda list
# packages in environment at C:\YourInstallFolder\bin\Python\envs\arcgispro-yourclone:
#

Python Package Manager—ArcGIS Pro | Documentation

[John_Spence]

Oh yeah...for sure switching environments, etc.  I had the same thought a couple of times where I was wondering if I was just seeing things.  To further answer your questions...

I made 2 different attempts to get the info you requested and/or get Conda to respond like it normally would.  In both cases, it the command "conda list" were not recognized commands.  Now you and I both know that shouldn't be the case.  So I tested it against a VM that I built from the ground up last night to verify/validate.

So...against the work PC running Windows 10 Enterprise 19041.985, I checked the following:

• Conda list | "'conda' is not recognized as an internal or external command, operable program or batch file"
• python.exe -m pip list returns everything you would expect.
• Visual Studio 2019 recognizes that the python version is a Conda environment.
• ArcGIS Pro internal work correctly (ie. notebooks, command line, etc.)

Against the home PC running Windows 10 Pro for Workstations 19042.985, I got the same exact results.  Conda was not recognized, but everything else worked as expected.  For the sake of sanity, I attached a snapshot of both the ArcGIS Pro and Python windows side by side.  

Oh...last but not least, before I even posted about this, I did uninstall and reinstall the software from the ground up on both machines.  I even cleared out the appdata, etc to make sure there were no remnants and that I was starting on a fresh sheet of music.

[Brian_Wilson · Accepted solution]

Conda is in a separate "scripts" directory so you have to make sure your PATH has that in it,  mine has "C:\Program Files\ArcGIS\Pro\bin\Python\Scripts"

Google "set windows 10 path" if you don't already know how to do that. 

If your "conda" command is not working then you are installing the package via pip into the wrong python environment.

It's possible to use "pip install packagename" once you have the right conda environment activated but you usually can just use "conda install packagename" instead. Personally I only use pip when conda install fails because using two different package managers gets confusing.

[John_Spence]

And the brain fog is lifted.....

Good call on shifting to the conda path.  I was able to resolve the issue I was running into.

I ended up uninstalling via pip the packages I installed and switched to installing them through the command line via Conda and it worked like a champ.

Frankly, everything was looking like a nail last night so using pip was the hammer that worked.  LOL.

Thanks a million to both you and @DanPatterson 

[DanPatterson]

proenv.bat is the file that you want to run.

Create a desktop shortcut if you want

Target : C:\...YourInstallFolder...\bin\Python\Scripts\proenv.bat

Start in :  C:\...YourInstallFolder...\bin\Python\envs\arcgispro-py3

That will start you in the main ArcGIS Pro environment.... change environments from there... and do install from whatever environment you want.

I don't use the package manager and I don't use clones, I install everything I need in the base environment  (except for the de rigueur packages like deep learning and its kin)

[John_Spence]

Always good to know how other folks are doing things.  I have been using the cloned environment for a while while doing development work.  Cloned environments are a pain because each time you upgrade, you need to rebuild the cloned environment and then...poof...issues like what I posted pop up where things are not working exactly as they had been before.  I am going to give some serious evaluation to your method as I work on other projects down the line.

Always a learning process in this field.  Got to love it.

[Brian_Wilson]

You can't install into the base unless you have admin rights on the machine, so cloning is a good work around for that.

I know it's yet another thing to remember when upgrading but you can save and reimport environments. See "conda list --help"

Save packages for future use:
    conda list --export > package-list.txt
Reinstall packages from an export file:
    conda create -n myenv --file package-list.txt

[Shingo-Ikeda]

Hi All, 

That's what happened to me yesterday by upgrading to 2.8 without keeping a backup python environment with deep learning packages. I installed DL package from the *.msi installer previously, but there is no 2.8 version of the installer out yet.  Once I created and activated a new environment, I installed DL package by executing the following command:

conda install -c esri deep-learning-essentials

This installed all, but gave me some errors at the end:

Preparing transaction: done
Verifying transaction: -
SafetyError: The package for fastai located at C:\Users\G0003563\AppData\Local\ESRI\conda\pkgs\fastai-1.0.60-py37_0
appears to be corrupted. The path 'Lib/site-packages/fastai/basic_train.py'
has an incorrect size.
  reported size: 31850 bytes
  actual size: 31851 bytes

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: esri/win-64::torch-cluster-1.5.4-py37_1, esri/win-64::torch-scatter-2.0.4-py37_2, esri/win-64::torch-spline-conv-1.2.0-py37_1, esri/win-64::torch-sparse-0.6.1-py37_1
  path: 'lib/site-packages/test/__init__.py'

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: esri/win-64::torch-cluster-1.5.4-py37_1, esri/win-64::torch-scatter-2.0.4-py37_2, esri/win-64::torch-spline-conv-1.2.0-py37_1, esri/win-64::torch-sparse-0.6.1-py37_1
  path: 'lib/site-packages/test/__pycache__/__init__.cpython-37.pyc'

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: esri/win-64::torch-cluster-1.5.4-py37_1, esri/win-64::torch-scatter-2.0.4-py37_2, esri/win-64::torch-spline-conv-1.2.0-py37_1, esri/win-64::torch-sparse-0.6.1-py37_1
  path: 'lib/site-packages/test/__pycache__/utils.cpython-37.pyc'

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: esri/win-64::torch-cluster-1.5.4-py37_1, esri/win-64::torch-scatter-2.0.4-py37_2, esri/win-64::torch-spline-conv-1.2.0-py37_1, esri/win-64::torch-sparse-0.6.1-py37_1
  path: 'lib/site-packages/test/utils.py'

done
Executing transaction: done

It looks like I can still use this environment and execute DL functions such as model.lr_find() and model.fit()

This could be just a warning, but I am not sure this is OK to ignore.

The link for the deep-learning-essentials is here:

https://anaconda.org/esri/deep-learning-essentials