topic Re: Installing conda prerequisites error in ArcGIS Pro Questions

Installing conda prerequisites error

MeshaSagram — Thu, 12 Sep 2019 16:57:31 GMT

Hello,

I am exploring the deep learning functions in ArcGIS Pro and have started following this tutorial: arcgis-python-api/detecting_swimming_pools_using_satellite_image_and_deep_learning.ipynb at master · Esri/arcgis-python-…

To get started with the arcgis.learn module in python, I need to install the prerequisites. The website says:

Data preparation, augmentation and model training workflows using arcgis.learn have a dependency on PyTorch and fast.ai deep learning libraries. They can be installed using conda as follows:
conda install -c fastai -c pytorch fastai=1.0.39 pytorch=1.0.0 torchvision

But when I enter that into my python window in Arc, I get a syntax error. I tried installing Anaconda and Git, and I entered this into the Anaconda Prompt and it seemed to install everything. But when I continue with the tutorial, I still get the message that it needs to be installed:

data = prepare_data(data_path, batch_size=32)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Mesha\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\learn\_data.py", line 132, in prepare_data
_raise_fastai_import_error()
File "C:\Users\Mesha\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\lib\site-packages\arcgis\learn\_data.py", line 22, in _raise_fastai_import_error
raise Exception('This module requires fastai, PyTorch and torchvision as its dependencies. Install it using "conda install -c pytorch -c fastai fastai=1.0.39 pytorch=1.0.0 torchvision"')
Exception: This module requires fastai, PyTorch and torchvision as its dependencies. Install it using "conda install -c pytorch -c fastai fastai=1.0.39 pytorch=1.0.0 torchvision"

Is there anyone who can tell me what I need to change so that this works? I can't seem to find any resources online that address this. Thank you!

Re: Installing conda prerequisites error

Anonymous User — Sun, 13 Oct 2019 16:32:38 GMT

Mesha Sagram‌

The best way to install these dependencies is documented here Install and set up | ArcGIS for Developers, this will install all the dependencies you need for deep learning. Further, if you need to train models you may also want to install jupyter notebook in the same environment, you need not install anaconda separately for that. I hope that helps and do reply here if you need some more information, also we have a wealth of samples like this one Land Cover Classification using Satellite Imagery and Deep Learning | ArcGIS for Developers .

Re: Installing conda prerequisites error

MeshaSagram — Fri, 15 Nov 2019 22:31:19 GMT

Hi Sandeep,

Thank you for your reply. I looked through the land cover classification and it seems very helpful. I opened the Python Command Prompt under ArcGIS and ran 'conda install -c esri arcgis' and 'conda install -c pytorch -c fastai fastai=1.0.39 pytorch=1.0.0 torchvision'. They both installed successfully so I continued with the tutorial. I tried using the python window in ArcGIS Pro, IDLE (Python GUI) and Jupyter notebook but I'm getting the same error. I also checked in ArcGIS Pro under installed packages and I have fastai 1.0.39, pytorch 1.0.0 and torchvision 0.2.2 installed. I'm not sure how to move forward and continue with the tutorial.

data = prepare_data(data_path, batch_size=16)

---------------------------------------------------------------------------Exception                                 Traceback (most recent call last)<ipython-input-4-ddf06bba82c5> in <module>----> 1 data = prepare_data(data_path, batch_size=16)C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\learn\_data.py in prepare_data(path, class_mapping, chip_size, val_split_pct, batch_size, transforms, collate_fn, seed, dataset_type)    110     111     if not HAS_FASTAI:--> 112         _raise_fastai_import_error()    113     114     if type(path) is str:C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\learn\_data.py in _raise_fastai_import_error()     16      17 def _raise_fastai_import_error():---> 18     raise Exception('This module requires fastai, PyTorch and torchvision as its dependencies. Install it using "conda install -c pytorch -c fastai fastai=1.0.39 pytorch=1.0.0 torchvision"')     19      20 def _bb_pad_collate(samples, pad_idx=0):Exception: This module requires fastai, PyTorch and torchvision as its dependencies. Install it using "conda install -c pytorch -c fastai fastai=1.0.39 pytorch=1.0.0 torchvision"

Re: Installing conda prerequisites error

Anonymous User — Sat, 16 Nov 2019 07:41:15 GMT

Hi mesha,

You can try installing arcgis with all deep learning libraries in a clean environment. Here is the documentation on creating a new environment Python Package Manager—ArcPy Get Started | ArcGIS Desktop, after that you can use this command 'conda install -c esri arcgis deep-learning-essentials --no-pin' in the python command prompt.

Re: Installing conda prerequisites error

Vishnu_maheshVivek_nanda — Tue, 14 Jan 2020 03:27:30 GMT

The command 'conda install -c esri arcgis deep-learning-essentials --no-pin' solved my dependency issues for the "Use Deep Learning to Assess Palm Tree Health" tutorial.U

Re: Installing conda prerequisites error

NicholasPiper1 — Fri, 17 Jan 2020 23:37:50 GMT

Sandeep,

It appears that arcgis deep-learning-essentials is no longer available through conda. Is there another location I can download it from?

-Nick

Re: Installing conda prerequisites error

Anonymous User — Sun, 19 Jan 2020 16:27:04 GMT

Hi Nick,

You can use this command now

conda install -c esri -c fastai -c pytorch arcgis fastai=1.0.54 pytorch=1.1.0 scikit-image --no-pin.

Thanks

Re: Installing conda prerequisites error

NicholasPiper — Mon, 20 Jan 2020 01:00:28 GMT

Sandeep,

Thank you for the reply, but this did not resolve my issue. I am still receiving this error:

Here is what I ran in conda and my ArcGIS Pro environment setting in case needed:

Any thoughts on where to go from here? Thank you again for your time!

-Nick

Re: Installing conda prerequisites error

Anonymous User — Mon, 20 Jan 2020 01:41:41 GMT

Nick, Please post the output of the command "conda list" from your environment, so that i can check the dependencies.

Re: Installing conda prerequisites error

NicholasPiper — Mon, 20 Jan 2020 02:15:10 GMT

(DeepLearningEnvTest) C:\Program Files\ArcGIS\Pro\bin\Python\envs\DeepLearningEnvTest>conda list
# packages in environment at C:\Program Files\ArcGIS\Pro\bin\Python\envs\DeepLearningEnvTest:
#
_tflow_select 2.1.0 gpu anaconda
absl-py 0.8.1 py36_0 anaconda
arcgis 1.7.0 py36_863 esri
arcgispro 2.4 0 esri
asn1crypto 0.24.0 py36_0
astor 0.8.0 py36_0 anaconda
atomicwrites 1.3.0 py36_1
attrs 19.1.0 py36_1
backcall 0.1.0 py36_0
beautifulsoup4 4.8.1 py36_0
blas 1.0 mkl
bleach 3.1.0 py36_0
bottleneck 1.2.1 py36h452e1ab_1
ca-certificates 2019.1.23 0
certifi 2019.3.9 py36_0
cffi 1.12.2 py36h7a1dbc1_1
cftime 1.0.0b1 py36h452e1ab_0
chardet 3.0.4 py36_1
cloudpickle 1.2.2 py_0 anaconda
cntk-gpu 2.7 <pip>
colorama 0.4.1 py36_0
cryptography 2.6.1 py36h7a1dbc1_0 esri
cudatoolkit 10.0.130 0 anaconda
cudnn 7.6.5 cuda10.0_0 anaconda
cycler 0.10.0 py36h009560c_0
cymem 2.0.2 py36h6538335_0 fastai
cython-blis 0.2.4 py36hfa6e2cd_1 fastai
cytoolz 0.10.0 py36he774522_0 anaconda
dask-core 2.9.0 py_0 anaconda
dataclasses 0.6 py_0 fastai
decorator 4.4.0 py36_1
defusedxml 0.5.0 py36_1
despatch 0.1.0 py36_0 esri
entrypoints 0.3 py36_0
et_xmlfile 1.0.1 py36h3d2d736_0
fastai 1.0.54 py36_0 esri
fastcache 1.0.2 py36hfa6e2cd_2
fastprogress 0.2.2 py_0 fastai
freetype 2.9.1 vc14_0 [vc14] esri
future 0.17.1 py36_0 esri
gast 0.3.2 py_0 anaconda
gdal 2.1.1 arcgispro_1 [arcgispro] esri
grpcio 1.12.1 py36h1a1b453_0 anaconda
h5py 2.9.0 py36_arcgispro_0 [arcgispro] esri
html5lib 1.0.1 py36_0
icc_rt 2019.0.0 arcgispro_0 [arcgispro] esri
idna 2.8 py36_0
imageio 2.6.1 py36_0 anaconda
intel-openmp 2019.3 arcgispro_203 [arcgispro] esri
ipykernel 5.1.0 py36h39e3cac_0
ipython 7.4.0 py36h39e3cac_0
ipython_genutils 0.2.0 py36h3c5d0ee_0
ipywidgets 7.4.2 py36_0
jdcal 1.4 py36_0
jedi 0.13.3 py36_0 esri
jinja2 2.10.1 py36_0
jpeg 9b hb83a4c4_2 anaconda
jsonschema 3.0.1 py36_0
jupyter_client 5.2.4 py36_0
jupyter_console 6.0.0 py36_0
jupyter_core 4.4.0 py36_0
jupyterlab 0.35.4 py36hf63ae98_0
jupyterlab_server 0.2.0 py36_0
keras 2.2.4 0 anaconda
keras-applications 1.0.8 py_0 anaconda
keras-base 2.2.4 py36_0 anaconda
keras-preprocessing 1.1.0 py_1 anaconda
keyring 19.0.1 py36_0 esri
kiwisolver 1.0.1 py36h6538335_0
libpng 1.6.36 h2a8f88b_0
libprotobuf 3.6.1 arcgispro_0 [arcgispro] esri
libsodium 1.0.16 h9d3ae62_0
libtiff 4.0.10 hb898794_2 anaconda
markdown 3.1.1 py36_0 anaconda
markupsafe 1.1.1 py36he774522_0
matplotlib 3.0.3 py36_0e [arcgispro] esri
mistune 0.8.4 py36he774522_0
mkl 2019.3 arcgispro_203 [arcgispro] esri
mkl_fft 1.0.10 py36h14836fe_0
mkl_random 1.0.2 py36h343c172_0
more-itertools 6.0.0 py36_0
mpmath 1.1.0 py36_0
murmurhash 1.0.2 py36h33f27b4_0
nbconvert 5.4.1 py36_0 esri
nbformat 4.4.0 py36h3a5bc1b_0
netcdf4 1.5.0.1 py36_arcgispro_1 [arcgispro] esri
networkx 2.4 py_0 anaconda
ninja 1.9.0 py36h74a9793_0
nose 1.3.7 py36_2
notebook 5.7.8 py36_0
numexpr 2.6.9 py36hdce8814_0
numpy 1.16.2 py36h19fb1c0_0
numpy-base 1.16.2 py36hc3f5095_0
nvidia-ml-py3 7.352.0 py_0 fastai
olefile 0.46 py_0 anaconda
openpyxl 2.6.1 py36_1
openssl 1.1.1b he774522_1
packaging 19.2 py_0
pandas 0.24.2 py36ha925a31_0
pandocfilters 1.4.2 py36_1
parso 0.3.4 py36_0
pickleshare 0.7.5 py36_0
pillow 6.1.0 py36hdc69c19_0 anaconda
pip 19.0.3 py36_0
plac 0.9.6 py36_0
pluggy 0.9.0 py36_0
preshed 2.0.1 py36h33f27b4_0
prometheus_client 0.6.0 py36_0 esri
prompt_toolkit 2.0.9 py36_0
protobuf 3.6.1 py36h33f27b4_0 anaconda
py 1.8.0 py36_0
pycparser 2.19 py36_0
pygments 2.3.1 py36_0
pyopenssl 19.0.0 py36_0
pyparsing 2.4.0 py_0
pypdf2 1.26.0 py_2 esri
pyrsistent 0.14.11 py36he774522_0
pyshp 1.2.12 py36_0
pysocks 1.6.8 py36_0
pytest 4.4.0 py36_1
python 3.6.8 h9f7ef89_7
python-dateutil 2.8.0 py36_0
pytorch 1.1.0 py3.6_cuda100_cudnn7_1 esri
pytz 2018.9 py36_0 esri
pywavelets 1.0.3 py36h8c2d366_1 anaconda
pywin32 223 py36hfa6e2cd_1 anaconda
pywin32-ctypes 0.2.0 py36he119be9_0 esri
pywinpty 0.5 py36_0 esri
pyyaml 3.12 py36_0 anaconda
pyzmq 18.0.0 py36ha925a31_0
requests 2.21.0 py36_0
scikit-image 0.15.0 py36_0 esri
scipy 1.2.1 py36h29ff71c_0
send2trash 1.5.0 py36_0
setuptools 41.0.0 py36_0
simplegeneric 0.8.1 py36_2
six 1.12.0 py36_0
soupsieve 1.9.5 py36_0
spacy 2.1.8 py36he980bc4_0 fastai
sqlite 3.27.2 he774522_0
srsly 0.1.0 py36h6538335_0 fastai
sympy 1.3 py36_0 esri
tensorboard 1.14.0 py36he3c9ec2_0 anaconda
tensorflow 1.14.0 gpu_py36h305fd99_0 anaconda
tensorflow-base 1.14.0 gpu_py36h55fc52a_0 anaconda
tensorflow-estimator 1.14.0 py_0 anaconda
tensorflow-gpu 1.14.0 h0d30ee6_0 anaconda
termcolor 1.1.0 py36_1 anaconda
terminado 0.8.1 py36_1
testpath 0.4.2 py36_0
thinc 7.0.8 py36he980bc4_0 fastai
tk 8.6.8 hfa6e2cd_0 anaconda
toolz 0.10.0 py_0 anaconda
torchvision 0.3.0 py36_cu100_1 pytorch
tornado 6.0.2 py36he774522_0
tqdm 4.40.2 py_0
traitlets 4.3.2 py36h096827d_0
typing 3.6.4 py36_0
urllib3 1.24.1 py36_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_1
wasabi 0.2.2 py_0 fastai
wcwidth 0.1.7 py36h3d5aa90_0
webencodings 0.5.1 py36_1
werkzeug 0.16.0 py_0 anaconda
wheel 0.33.1 py36_0
widgetsnbextension 3.4.2 py36_0
win_inet_pton 1.1.0 py36_0
wincertstore 0.2 py36h7fe50ca_0
winkerberos 0.7.0 py36_1
winpty 0.4.3 4
wrapt 1.11.2 py36he774522_0 anaconda
x86cpu 0.4 py36_1 esri
xlrd 1.2.0 py36_0
xlwt 1.3.0 py36h1a4751e_0
xz 5.2.4 h2fa13f4_4 anaconda
zeromq 4.3.1 h33f27b4_3
zlib 1.2.11 h62dcd97_3
zstd 1.3.7 h508b16e_0 anaconda

Re: Installing conda prerequisites error

TomThompson1 — Wed, 04 Mar 2020 14:59:22 GMT

Hey Sandeep Kumar,

I'm also seeing this issue with arcgis.learn. I've following the guide in Geospatial deep learning with | ArcGIS for Developers and installed the modules within a new environment successfully.

When running prepare_data() function from the learn module within Jupyter Notebooks, it throws the _raise_fastai_import_error(), claiming that it requires those modules listed in the original post. However, interestingly, when I run the _data.py file directly within VS Code, it succeeds and no exception is thrown.

I've confirmed that the environment I'm running is the same between my Jupyter Notebook & VS Code, AppData\Local\ESRI\conda\envs\arcgispro-py3-clone1. So I do not know why the exception occurs in Jupyter Notebooks, but does not in VS Code.

This problem I'm seeing may also be the root cause of why others experience this problem, so I'm posting here with these findings. I'll do some more debugging today and if I find the solution to my problem, I'll post a reply with the fix.

Tom Thompson, Esri Canada Technical Support.

Re: Installing conda prerequisites error

DanPatterson_Retired — Wed, 04 Mar 2020 15:09:28 GMT

did you try --no-pin in conda?

Re: Installing conda prerequisites error

TomThompson1 — Sat, 11 Dec 2021 16:17:11 GMT

Some progress on this. I believe this issue is related to the _data.py file. Something, at least on my machine, is causing the import statements within the try/except to stop working.

Specifically, it is the import statement:

from fastai.vision.data import imagenet_stats, ImageItemList

Any statement defined after this line is skipped and fails, and so an exception is thrown and HAS_FASTAI will always equal false.

Interestingly, the interpreter throws no errors. It just exits and appears to have executed successfully. My debugger wasn't working, so I implemented some print statements to test.

try:
 print("---_data.py Modules & Versions---")
 import fastai
 print("fastai installed at version: ",fastai.__version__)
 import torchvision
 print("torchvision installed at version: ",torchvision.__version__)
 import torch
 print("torch installed at version: ",torch.__version__)
 print("Before Import from fastai.vision.data import imagenet_stats, ImageItemList")
 from fastai.vision.data import imagenet_stats, ImageItemList
 print("After Import from fastai.vision.data import imagenet_stats, ImageItemList")‍‍‍‍‍‍‍‍‍‍‍

This is quite bizzare.

What do you guys think?

Tom

Re: Installing conda prerequisites error

TomThompson1 — Wed, 04 Mar 2020 16:00:15 GMT

Hi Dan Patterson‌,

I did try the --no-pin when installing the modules in my old environment and it did not resolve the problems.

I'll create a third environment and test it again just to make sure there were no dependency miss-matches.

Thanks for the suggestion,

Tom

Re: Installing conda prerequisites error

Anonymous User — Wed, 04 Mar 2020 16:25:59 GMT

Hi Tom,

I see you are using ArcGIS Pro python command prompt, Create a new cloned environment and install the deep learning dependencies using this command

conda install -c esri arcgis fastai pillow scikit-image --no-pin

Re: Installing conda prerequisites error

TomThompson1 — Wed, 04 Mar 2020 16:45:09 GMT

Hey Dan Patterson,

Tested using the --no-pin and the documented solution to installing these modules:

conda install -c fastai -c pytorch fastai=1.0.54 pytorch=1.1.0 torchvision --no-pin

This did not resolve the problem.

Great idea though, I was hopeful. Thanks again for the suggestion.

Tom

Re: Installing conda prerequisites error

DanPatterson_Retired — Wed, 04 Mar 2020 16:52:24 GMT

Sandeep is on to something... you can't to the installation from your command prompt, you have to be in the cloned conda environment

Re: Installing conda prerequisites error

TomThompson1 — Wed, 04 Mar 2020 16:58:52 GMT

Hi Sandeep Kumar,

I did as you suggested. Built a new clone and used the conda command you provided.

Same problems continue. Something to do with the import statements in the _data.py file. Occurs on the import of:

from fastai.vision.data import imagenet_stats, ImageItemList

Nothing is executed after this line in the _data.py and so my Python scripts leveraging the prepare_data() function fail, claiming that I do not have the required modules.

My next test is to update ArcGIS Pro to v2.5.0, as I'm at V2.4.3. Once updated I'll create another fresh clone and use the command you provided.

Tom

Re: Installing conda prerequisites error

Anonymous User — Wed, 04 Mar 2020 17:02:02 GMT

tom can you try this, after you import arcgis in the same notebook

import torch

also just to make sure ipython is working properly try this command in the python prompt

python -m ipykernel install --user

Re: Installing conda prerequisites error

TomThompson1 — Wed, 04 Mar 2020 17:13:09 GMT

I've been installing all modules through the Python Command Prompt. That screen capture is of VS Codes terminal, so it is a bit misleading.