How do you get Python packages installed with conda/pip to show up in the Python Package Manager?

Question

I need access to Azure 4.0.0. There is (of course) a pip install available, but no conda install (highest version for conda install is 1.0.2). Accordingly, I created a cloned environment, and installed from the command line via conda and pip:

activate arcgispro-py3-azure-crypto (my cloned environment; pip is already in this environment, however, I did upgrade pip to 19.1.1.)

pip install azure

When I list packages in conda, everything looks fine:

However, when I look at the environment in Pro, nothing is there:

Everything is working, which means the conda listing is correct (and the packages are installed in the appropriate envs directory, but not in the pkgs directory of <localappdata>\ESRI\conda), but it’s just not showing up in the Pro Python Package Manager. I did a little digging around, and it looks like the Python Package Manager is actually reading the above contents from C:\Program Files\ArcGIS\Pro\bin\Python\pkg-metadata\metadata.json.

* Question 1: How do I get these pip-installed packages to show up in the Python Package Manager?
* Question 2: Is the Python Package Manager reading from the metadata.json file, and if so, how do we update it?
* Question 3 (and this is a larger question): What is the best practice for installing our stuff in conda world? Are we OK to install our own packages directly to the arcgispro-py3 environment, or should we be creating a new environment (or selecting another existing environment) and installing to that?

Thanks in advance!

Tracy

TracyThorleifson · Answer

Dan,

Yes, we currently install to the base environment, too (thus our installers require admin privileges). Actually, this whole thread came up as a result of one of our Python toolbox tools throwing an error on my machine because YT was attempting to run in an environment where I had not installed the underlying site package. (Causing me to realize our Python toolbox tools need to trap for that, too. Ah, the brave new world of conda. Grr.) As I dug into it, I realized that none of our pip-installed dependency packages were showing up at all. So there you go.

With respect to our installers, it occurs to me that it might be proper to actually present the user with a checkbox list of the existing environments, and prompt them to select the environments in which they want our software to be installed, and/or specify a new environment in which to install the software. If the user lacks admin privileges, we could exclude the base environment as a choice. Of course, all of this goodness would complicate our installers considerably, which is why I'm seeking feedback from the Esri poobahs. If Esri would rather we not install into the base environment at all, we could make that happen.

Best regards,

T.

DanPatterson_Retired · Answer

I download the *.exe, unzip in a folder keep the *.msi and *.cab.  Worst comes to worse, wipe ArcAnything from the machine and reinstall. The package manager is for clones.  I use conda and install in the base environment

Perhaps this protracted discussion between Cheryl and I as we parachuted on another post might be of interest

https://community.esri.com/thread/226754-trouble-installing-spyder#comment-856644

As for 'official' package installers, there is official and officialish and GitHub and whatever.  Nothing guarantees things will work well under all circumstances

TracyThorleifson · Answer

Dan,

Thank you for replying so promptly. I actually checked out your posts before I posted. 🙂 Anyway, conda installs work just fine; conda installs do show up in the Package Manager. It's pip installs that are giving me fits. Yes, I know, conda and pip don't always play well together, and we need conda to deal with non-Python dependencies, but pip is the official Python installer, and conda installs often lag pip installs by several versions. That's my problem: I need the latest and greatest from Azure, and the available conda install doesn't cut it. As I mentioned above, the packages are there and working properly, they just don't show up in the Package Manager (which is then a bit misleading to the user, eh?).

With respect to best practices for installing our own software, what I'm really looking for is some guidance from Esri. We've always installed into the default Esri Pro Python environment, but at 2.2, even users with admin privileges can't do so through the Package Manager GUI. (You, can of course, have your way with things at the command line.) So, if I'm installing my own Python site package, should I do so only in a cloned environment, or not, as a matter of best practice?

Thanks again,

T.