Deep Learning - data.show - depleted VRAM

6675
18
Jump to solution
08-31-2020 07:23 AM
Labels (2)
VHolubec
Esri Regular Contributor

Dear Imagery and Remote Sensing,

I am trying the Esri example - https://developers.arcgis.com/python/sample-notebooks/automate-building-footprint-extraction-using-i... , on the Czech data.

I´ve stuck on the beginning with the command data.show_batch(rows=4).

I am getting a error:

CUDA out of memory. Tried to allocate 88.00 MiB (GPU 0; 4.00 GiB total capacity; 483.95 MiB already allocated; 64.31 MiB free; 500.00 MiB reserved in total by PyTorch)

 

My GPU has 4GB of VRAM and almost 75% is allocated by the data.show command. And it´s still allocated. even after the result is displayed. Deleting of the Cell did not help.

 

I would like to ask you for help in three questions:

 

1) Is there a way how to free VRAM after displaying data.show_batch(rows=4).

2) Is there a way how to use RAM to help the VRAM?

3) What is the recommended VRAM size?

 

Thank you,

 

Vladimir

Vladimir
0 Kudos
18 Replies
PriyankaTuteja
Esri Contributor

Hi @VHolubec ,

If you want to use a better GPU for training a deep learning model, we provide ArcGIS Notebook Server with a ready to use deep learning environment. 

PriyankaTuteja_0-1614145096559.png

Credentials: https://geosaurus.maps.arcgis.com/home , arcgis_python, P@ssword123

You need to create a notebook with 'Advanced with GPU support' option as shown in the screenshot. 

Now coming to how will you get your exported data on this portal, please refer to https://developers.arcgis.com/python/sample-notebooks/vehicle-detection-and-tracking/ or any other deep learning notebook available. You can do the following steps:

1. zip your training data and upload it as image collection item. For that, click on Contents> Add item>from your computer. Then upload zip file and change the type to image collection. 

2. Follow the steps from the notebook link mentioned above and unzip your data and complete the rest of the workflow to train a model. When you get a good model, call save('name', publish=True). This will publish the dlpk as an item and you can download it locally to your disk.

Let me know if there's anything we could help with.

 

Regards,

Priyanka

VHolubec
Esri Regular Contributor

Dear @PriyankaTuteja ,

that´s awesome. Thank you very much! 

Have a great day,

Vladimir

Vladimir
NatalieDobbs1
New Contributor II

Hi @PriyankaTuteja 

Do you have any feedback concerning @VHolubec questions above 

Thanks

Natalie

by Anonymous User
Not applicable

Hey friend, this is somewhat unrelated to your question, but I have been having issues trying to get "data.show_batch()" to do anything at all.  I've tried adding row parameters to it, all prior code leading to it looks solid, etc etc..  I just cannot get anything to display, particularly samples of the data, which I need to see.  

Just wondering if you could briefly tell me about how it works.  I'm new to notebooks as well so I don't know if it is supposed to just display the samples automatically, or you must make a special cell to have them displayed, or what.  Tutorials taunt me with what to expect, but no explanation of what to do if it doesn't work.  

0 Kudos
VHolubec
Esri Regular Contributor

@Anonymous User  can you share your code here? 

Vladimir
by Anonymous User
Not applicable

So I just got the batch data to show in a jupyter notebook in browser (via importing arcgis), which is certainly progress.  Apparently most experienced people do it that way anyway.

However, the purpose of my project is to do GIS deep learning techniques (particularly object detection) through as many means as I can.  Think of it as I am presenting to faculty different ways to approach this to determine the best way to teach it to future students.  Thus, I kinda need to get this running inside of the ArcGIS Pro notebook itself (inside of the app).  

I believe that I have narrowed that down to versioning issues with python packages.  I tried dealing with that the other day, however the package manager inside of ArcGIS Pro wouldn't update anything.  I tried updating through the python command prompt, but in order to do the updates it for whatever reason forced me to accept downgrades to some packages, which bricked the notebook entirely (good thing I kept the default environment in virgin state).  

So that is kind of where I am with that.  Not quite my original inquiry, but if you know a better way to update all packages in my environment, please let me know.  Additionally, I would like to know if there are actual up-to-date (deep learning) environment files available for download from like github or whatever that I can just copy & paste as a new environment.  

PriyankaTuteja
Esri Contributor

Hi @Anonymous User ,

 

I am glad to hear that you want to apply different techniques to accomplish object detection task integrated with GIS. I would like to offer help whenever you get stuck. 

Here are some initial questions for you:

1. What ArcGIS Pro version do you have?

2. How have you created your deep learning environment as of now?

 

A general guide for ArcGIS Pro version 2.7:

Set up deep learning environment in ArcGIS Pro 2.7: https://github.com/Esri/deep-learning-frameworks 

The default environment of ArcGIS Pro 2.7 version comes with arcgis version 1.8.4. However, given your interest in running the deep learning workflow inside ArcGIS Pro  embedded notebooks, I would suggest cloning the default environment and upgrade to 1.8.5 version of arcgis. 1.8.5 version has a bug fixed particularly to make embedded notebooks in ArcGIS Pro work fine! For details see this.

 

Thanks,

Priyanka

0 Kudos
by Anonymous User
Not applicable

packagesUpdate.pngHello, so I am using ArcGIS Pro 2.7.2.  I have the deep learning packages installed correctly to the default vanilla environment in AGP (so I can clone it to make changes).  

The main reason why I would use the notebook for the AGP workflow is due to the vanilla training model tool simply lacking compatibility with the demands of the data.  I am at this moment training a model using jupyter notebook in browser while importing arcgis learn module.  So I am getting it to work, with this work around, though of course I want to develop a repeatable workflow for just using the internal AGP notebook.  

So I am about 100% sure that the reason why the AGP notebook was not working correctly (ie., displaying batches), was due to some packages requiring updates- 41 in total, apparently (see attached image).  However, as I mentioned before, I cannot seem to update any of them inside of the AGP options interface at all.  Like I click "update" and it does nothing.  I've tried doing that and restarting it, and nothing changes.  This is with a cloned environment where it should work.  I've also tried changing permissions on the cloned environment file itself, but it seems to be at highest level already.  I also tried updating it within the python command shell, but while it says it will do all of the package updates, it also requires a few package downgrades- one of which was the jupyter server or whatever, which crashed notebooks inside of AGP.  

So if I understand you correctly, are you saying that I could just update a specific package in the cloned environment using the python command shell?  That may work, just seems really inefficient.  The part that I don't understand is why is required the downgrade for some packages, which seemed odd.  

EDIT: just to clarify, arcgis 1.8.5 is installed in my environments.  

0 Kudos
VHolubec
Esri Regular Contributor

Hello @Anonymous User ,

so just to understand your status.

1) You have created a cloned Python environment in ArcGIS PRO

2) You have set it up as default

3) You have installed there some other python packages

Is that correct? 

To be honest I don´t use the PRO for managing the Python. So I can give you an advice to manage it with Python Command prompt.

Open Python Command prompt

VHolubec_0-1620116680728.png

 

There verify the environment which is used - in my case it´s cloned environment called DL

VHolubec_1-1620116732024.png

- to install a package run: conda install <package name>

- to install a package with a specific version run:  conda install <package name>=<version>

- to update a specific package run: conda update <package name>

- to update all run: conda update --all 

The last command will update all packages in the current environment to the latest version, but it may use an older version of some packages in order to satisfy dependency constraints.


Hope this helps.

Vladimir

 

Vladimir
0 Kudos