topic Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8 in ArcGIS API for Python Questions

Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

JackCharde1 — Tue, 16 Feb 2021 05:20:06 GMT

Hey everyone (and @KarthikDutt, who asked me to post this here!)

I ran into an issue while using Jupyter Notebook and arcgis.learn for the first time while following this Esri Youtube Video

I found the github location for the workflow and tried following the documentation but ran into dependency installation issues and deep learning tool model existence in certain versions of Anaconda.

I could not get this to work in a Jupyter Notebook. Maybe someone might have some insight!

I downloaded Anaconda, version 3.8, and installed ArcGIS API for Python (arcgis) per the blog "Three Ways to Get Jupyter Notebooks and the ArcGIS API for Python".

With my data in a notebook, I tried according to the video at timestamp 2:07:

ln[1]: from arcgis.learn import prepare_data, MultiTaskRoadExtractor

ln[2]: data_path=..... <<(I selected the path where my image chips were stored)

ln[3]: data = prepare_data(data_path, chip_size=256, batch_size=135)

I received a traceback error on ln[3] saying I was missing a number of dependencies required to use MultiTaskRoadExtractor. So, I opened the Anaconda command prompt and tried installing them, only to be told that there were a bunch of conflicts that prevented install.

So I researched how to create new Anaconda environments, and created one for Python 3.6, because some forums said some dependencies like earlier versions better. I installed arcgis so I could access Jupyter Notebooks in Python 3.6. Also, in my Python 3.6 environment, the dependencies now installed without issue. But then when I tried the above code again in a 3.6 Notebook, ln[1] returned a traceback error saying:

"ImportError: cannot import name 'MultiTaskRoadExtractor' from 'arcgis.learn' (C:\Users\JohnC\anaconda3\envs\snakes\lib\site-packages\arcgis\learn\__init__.py)"

To see if there was a model existence issue, I compared the 'lib\site-packages\arcgis\learn\__init__.py" files between environments, and the Python 3.6 environment did not have the MultiTaskRoadExtractor model. Version 3.8 does.

So I tried creating another anaconda environment in Version 3.7 to see if I could install these model dependencies and see if the MultiTaskRoadExtractor model was available for import into a Notebook. It also was not. It seems only Version 3.8 has this model, but the dependencies also won't download to the 3.8 environment.

So I'm unsure what to do, and I've exhausted my level of knowledge with this stuff (which is next to nothing).

Do you have any suggestions? Is there more information you need from me to answer this? Let me know and thank you in advance.

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

DanPatterson — Sun, 14 Feb 2021 03:51:16 GMT

How Multi-Task Road Extractor works ? | ArcGIS for Developers

Did you check the following section in that link?

Setting up the environment

You might want to check out the github site as well

Esri/deep-learning-frameworks: Installation support for Deep Learning Frameworks for the ArcGIS System (github.com)

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

JackCharde1 — Sun, 14 Feb 2021 05:06:54 GMT

@DanPatterson I did look at the first link, but I will look over it again (maybe something will click in my brain) and definitely check out the 2nd.

Thanks!

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

JackCharde1 — Wed, 17 Feb 2021 17:14:12 GMT

So, I got it to work, but only when using a notebook within ArcGIS Pro. After installing the Deep Learning Frameworks to my ArcGIS Pro install, I was able to use the MultiTaskRoadExtractor model within the program, because all of the necessary libraries were finally available.

Trying to use the model in a Jupyter Notebook through a web browser tab, activated by my Anaconda Command Prompt still does not work, because of the weird dependency install issues.

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

DanPatterson — Wed, 17 Feb 2021 21:51:06 GMT

whatever environment is activated through your conda prompt would also have to have all the dependencies installed.

Deep learning .... makes you wonder sometimes 😉

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

JackCharde1 — Wed, 17 Feb 2021 23:02:20 GMT

That was the issue. I tried installing the dependencies on three environments: default (v3.8), "rattler" (v3.6), and "snakes" (3.7). They installed on 3.6 and 3.7 without issue, but wouldn't install on 3.8. The arcgis package installed on each version without issue, but versions 3.6 and 3.7 didn't include the MultiTaskRoadExtractor model, so I couldn't use those versions to run a jupyter notebook. So my only option was my 3.8 environment, which wouldn't install the dependencies.

Haha, deep learning indeed. Maybe one day it can deep learn my issues and solve 'em for me!

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

DanPatterson — Wed, 17 Feb 2021 23:23:57 GMT

some of the deep learning packages aren't/weren't supported in python 3.8 (tensorflow is one if memory serves).

I suspect they have been slowing up arcpy upgrades which is why 3.7 is the version for arcgis pro 2.7.

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

JackCharde1 — Thu, 18 Feb 2021 16:54:43 GMT

That definitely seems like the issue. Thanks for the help!

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

Anonymous User — Fri, 19 Feb 2021 03:23:48 GMT

Hi Jack,

MultiTaskRoadExtractor model was added in ArcGIS Python API version 1.8.1, any subsequent release will have that. Can you also attach output of conda list from your env where it is not working.

Thanks,

Sandeep

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

JackCharde1 — Sat, 20 Feb 2021 20:09:18 GMT

@Anonymous User hopefully the attached photos are what you are asking for! I am so new to this, and constantly researching when I run into something new I've not done before.

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

JackCharde1 — Mon, 08 Mar 2021 06:13:08 GMT

And here I've attached snippets of each of the environments arcgis.learn __init__.py files to show that only v3.8 inlcudes MultiTaskRoadExtractor. v3.6 and v3.7 don't. Maybe there's something I'm missing?

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

Anonymous User — Fri, 26 Feb 2021 04:36:37 GMT

I tried creating a new environment and it worked fine for me, I will suggest you to try this:

1. Clean existing conda packages by the following command

`conda clean -a`

2. Create a new python 3.7 environment and try installing arcgis with all deep learning dependencies there.

Make sure the latest version of the Python API for ArcGIS gets installed, for me it was 1.8.4.

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

JackCharde1 — Sat, 27 Feb 2021 19:37:32 GMT

@Anonymous User I just attempted this, per your instruction.

I created a fresh 3.7 environment in conda, installed arcgis (it was 1.8.4, like yours), and then installed the dependencies, according to the very bottom of this link.

The attachment "Pre_DL_..." shows that the MultiTaskRoadExtractor model is present when installing arcgis 1.8.4.

When I went to install the deep learning dependencies, conda said that arcgis would be downgraded, from 1.8.4 --> 1.8.2 (Attachment: "arcgis_will_down..."). I said yes, and it installed the dependencies.

I went back to the __init__.py file after I installed the dependencies, and from my computer's File Explorer, it shows that MultiTaskRoadExtractor is still available (Attachment: "Post_DL_...").

However, when I use conda to open Jupyter Notebook in my browser, and run the code:

ln[1]: from arcgis.learn import prepare_data, MultiTaskRoadExtractor

I get a traceback error saying it cannot import the model. Searching for the arcgis.learn __init__.py file through the jupyter notebook directory, I find that the model is not available (Attachment: "JupyterNotebook__...").

Now I should mention that I can open a notebook from within ArcGIS Pro and get all of this to work on the base environment. But it just doesn't want to work in an external 3.7 environment through Jupyter Notebook.

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

Anonymous User — Mon, 08 Mar 2021 05:06:20 GMT

Hi @JackCharde1

Can you replace the arcgis API version in the installation command and try again.

conda install -c esri -c fastai -c pytorch arcgis=1.8.4 scikit-image=0.15.0 pillow=6.2.2 libtiff=4.0.10 fastai=1.0.60 pytorch=1.4.0 torchvision=0.5.0 --no-pin

Thanks,

Sandeep

Re: Errors with dependency installation and Deep Learning Tool model availability in Anaconda 3.8

JackCharde1 — Mon, 08 Mar 2021 06:22:33 GMT

That worked! Thank you so much @Anonymous User !

I just checked it in a browser-based notebook and encountered no errors running the successive lines of code.

Next step: get more CUDA!

Cheers 👍🍻