Conda... the dependency trail

1391
1
08-12-2020 07:58 PM
Labels (1)
DanPatterson
MVP Esteemed Contributor
2 1 1,391

C onda

Packages.

Stuff depends on other stuff.  A brief summary of packages and package management.

Clone.

Supposedly keeps stuff from getting muddled when you import the wrong version of a package which causes another package to complain loudly.

Versions

So you want the latest and greatest... maybe the package version causes problems with other packages.

 

Python 3.6.10... why is ArcGIS Pro using that version?  Laziness? Not likely.  More probably a required package doesn't work with python 3.7 or 3.8.

 

Some packages support other versions of python, for instance, but not all do.  Some packages are a bit slower to move with forward.  Some packages move in step with other packages so multiple updates will occur in related packages at the same time.

 

How do you keep track of all this stuff?  That is what conda is supposed to be used for.

How many deep learning threads have you seen on GeoNet where there are issues?  Most related to outdated versions of Pro, or a version of Pro not synching with the correct version of the required package, or a packaged is upgraded to the latest but you haven't upgrades the support framework.  Stick with simple learning first.

 

To add to the muddle, how about a little script that allows you to produce 3 tables in Pro so that you can examine the interdependencies of packages in your conda environment.  There are variants of how to examine the dependency tree I will present one that reads the *.json files in your conda-meta folder (bet you didn't know you had one).

 

Here is what you can get

 

Package, filename and dependencies

 

The Filename column could be split out on the "-" should you wish to determine the installed version and where it was downloaded from.

The Dependencies field either has None, a single package name, or a semicolon delimited list of needed packages.

 

 

Package and required by

 

Now this can be a bit confusing since it appears some packages require Pro to work... they don't, they are just package versions used by the Package.  But for the most part, you can cruise through the list trying to figure out what depends on what and how many things may depend on the same package.  A useful eye-glazer.

 

Package dependency counts

To make it simple to read, I will just show the sorted table of the dependencies in descending order.  Who knew that vc was top of the list.  And surprise! 33 need python, but wait, there is 23 and 7 required for variants of python.  This enables you to fine-tune dependency version.  Amazingly, not all packages indicate that they are dependent on python even though they are written in python.  Hmmm I wonder how many other things leave out dependencies from their list.

 

 

Spyder

My favorite Python IDE and my go-to for Notebooks when I don't need the webby versions or the Pro version.

Seemingly a lot of dependencies, but many are shared with variants of notebooks or are related to formatting, fonts, syntax highlighting, code completion and linting (aka... cool coding stuff )

 

 

So give  pkg_info_json(folder=None) a run.

 

Eventually I will get around to mapping this dependency space using networkx as suggested by a couple of blogs.  Could be fun.

 

So next time you want to mess with your clone or you main/base, see if you can unpackage what you have done.  That is why there is the ability to go back in time to a simpler say with

conda install --revision 1‍

The number 1 being replaced with the pre-goof state.

 

Have fun, be safe.

1 Comment
JoeBorgione
MVP Emeritus

Interesting read Dan.  Thanks!

About the Author
Retired Geomatics Instructor (also DanPatterson_Retired). Currently working on geometry projects (various) as they relate to GIS and spatial analysis. I use NumPy, python and kin and interface with ArcGIS Pro.
Labels