topic Re: Detect Objects Using Deep Learning Error with new RTX 3060 in ArcGIS Pro Questions

Detect Objects Using Deep Learning Error with new RTX 3060

Hulseyj — Thu, 22 Apr 2021 02:09:11 GMT

Error appears when I use "Detect Objects Using Deep Learning" in ArcGIS Pro with new RTX3060 GPU. The model used is usa_building_footprints.dlpk. I am using ArcGIS Pro 2.7.3. I have no issues running the model with my GTX 1650 Super GPU, same computer, same settings. It appears ArcPro engages the GPU as the application is shown running on the GPU:

However, the program shows as "Running" but never progresses past 0%.

After about 30 minutes of "running", the following error message is returned:

I used the Input Parameters and Environment below. Again, it runs fine with the GTX 1650 Super GPU. All drivers are up to date on the new RTX3060.

Thank you for help or troubleshooting ideas.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

DanPatterson — Thu, 22 Apr 2021 01:59:15 GMT

recent blogs on this and related topics especially since your hardware is new

Deep Learning with ArcGIS Pro Tips & Tricks: Part 1 (esri.com)

Deep Learning with ArcGIS Pro Tips & Tricks: Part 2 (esri.com)

Re: Detect Objects Using Deep Learning Error with new RTX 3060

Hulseyj — Thu, 22 Apr 2021 03:44:19 GMT

Thanks, Dan. I have worked through the recommended steps in the blogs. The CUDA Toolkit has been installed, and running deviceQuery show both cards are CUDA-capable and detected.

BandwidthTest passed:

The Deep Learning Libraries for Arc Pro have been installed. I am using the default arcgispro-py3 environment.

I am not sure what else to try. The out-of-box model works fine with the GTX 1650 Super, but not with the new RTX 3060.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

Tim_McGinnes — Thu, 22 Apr 2021 05:00:36 GMT

An issue was just posted to Esri's deep learning GitHub site about this (may be caused by a lack of CUDA 11 support in the underlying packages):

https://github.com/Esri/deep-learning-frameworks/issues/17

Probably worth adding your name to that as well - the more people raising it the more likely it might get fixed.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

AndrewEastop — Thu, 22 Apr 2021 11:59:35 GMT

Hi Hulseyj, I have just experienced the same problem as yourself. Did you find a resolution?

Re: Detect Objects Using Deep Learning Error with new RTX 3060

Hulseyj — Fri, 23 Apr 2021 00:04:13 GMT

Thanks Tim. I added my name and issue to that post on GitHub.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

Hulseyj — Fri, 23 Apr 2021 00:06:39 GMT

Thanks Andrew. Glad to know I am not the only one experiencing the issue. No resolution yet.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

AndrewEastop — Fri, 23 Apr 2021 07:55:48 GMT

Have you tried downloading this?

Install Deep Learning Frameworks for the ArcGIS System

Re: Detect Objects Using Deep Learning Error with new RTX 3060

_Cartographer_ — Fri, 23 Apr 2021 16:56:23 GMT

This is 100% caused by lack of CUDA 11 support and lack of updated libraries for PyTorch, TensorFlow, etc.

The latest RTX 30 series GPUs require CUDA 11.

ESRI's current "Deep Learning Frameworks" as linked above use CUDA 10.1

Unfortunately, looking at the pre-release frameworks for ArcGIS Pro 2.8 (available here: https://anaconda.org/Esri/deep-learning-essentials/files) they have still not updated to CUDA 11, or updated PyTorch/TensorFlow. I do not understand why not. I think this comes down to demand.

Everyone who has this issue should be posting and making sure that ESRI knows this is a problem for an increasing number of users -- and will obviously only increase further as more people get the latest GPUs. With how much ESRI is touting ArcGIS deep learning capabilities, this slow response is quite surprising.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

NathanPamperin1 — Fri, 23 Apr 2021 22:33:47 GMT

Getting the same error (below) trying to train deep learning models in some instances. I am running an Nvidia Geforce RTX 3080 (laptop). I can get the same models trained on a desktop Quadro RTX 5000 that has CUDA toolkit 11.2 installed but also has an instance of CUDA 10.1 installed as well. For what its worth...

File "C:\Users\natep\AppData\Local\ESRI\conda\envs\DeepLearning\Lib\site-packages\torchvision\ops\boxes.py", line 95, in remove_small_boxes
keep = keep.nonzero().squeeze(1)
RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered
Failed to execute (TrainDeepLearningModel).

Re: Detect Objects Using Deep Learning Error with new RTX 3060

Samaloysius — Mon, 26 Apr 2021 23:04:04 GMT

Hi everyone ,

Thank you for your comments. Due to potential breaking changes, For ArcGIS Pro 2.8, the versions have not been updated. We are looking into upgrading the versions of CUDA , PyTorch, TensorFlow for the next release.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

NathanPamperin1 — Tue, 27 Apr 2021 15:24:13 GMT

So that would be ~November for Pro v2.9?

Re: Detect Objects Using Deep Learning Error with new RTX 3060

Samaloysius — Tue, 27 Apr 2021 16:05:28 GMT

That is the timeframe for Pro v2.9 , but are looking into ways to possibly provide support for this earlier. Apologies for the inconvenience.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

NathanPamperin1 — Tue, 27 Apr 2021 16:54:25 GMT

Thanks for the info and I appreciate the efforts to try to come up with a solution ahead of v2.9!

Re: Detect Objects Using Deep Learning Error with new RTX 3060

_Cartographer_ — Wed, 05 May 2021 04:40:44 GMT

Any possible support before v2.9 would be seriously appreciated. That's a long time to wait -- and many more users will encounter this problem in the meantime. I have already volunteered to be an early adopter to help test if needed -- at least then I could use my GPU as originally intended. I'm sure a number of us would be happy to alpha/beta test if it meant getting this issue moving forward.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

NathanPamperin1 — Wed, 05 May 2021 04:49:00 GMT

I’d be happy to test as well!

Re: Detect Objects Using Deep Learning Error with new RTX 3060

Samaloysius — Wed, 05 May 2021 19:46:24 GMT

Thank you . We are working on it.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

Samaloysius — Wed, 05 May 2021 19:47:42 GMT

Thank you Nathan!

Re: Detect Objects Using Deep Learning Error with new RTX 3060

Anonymous User — Fri, 07 May 2021 05:32:27 GMT

Hey, so I was referred here from GitHub. Same story for me with RTX 3080. Using the Imagery Analyst DL toolset for me in ArcGIS Pro usually results in waiting 30 minutes for mysterious CUDA errors or false successes. Anyway, here is a list of issues I have encountered, some of which may not be directly related to GPU, but the "CUDA Assert" and "NaN" errors (resulting from training epochs) are big problems:

Just for the record, issues I've run into with my RTX 3080:
- Many "CUDA Assert" errors throughout my attempts to use tools in ArcGIS Pro.
- Create image chips tool refusing to process RCNN files for no apparent reason; PASCALs worked.
- False success with DL tools that claim to have been successful, but nothing whatsoever results from running the final model, even just to do a basic test for any result.
- Certain elements in ArcGIS Notebook not working at all, like show batch samples, which work in external Jupyter Notebook (though this could be unrelated to GPU).
- In external Jupyter Notebook/Training Model: Optimal Learning Rate graph feels buggy.
- In external Jupyter Notebook/Training Model: Training the model in any instance results in "NaN" values per epoch statistics, which is probably linked to previous point where ArcGIS DL Tools were "successful" but nothing meaningful was ever produced.

Re: Detect Objects Using Deep Learning Error with new RTX 3060

samaloysius4 — Thu, 13 May 2021 21:59:44 GMT

Hi All,

Thank you for providing details on your workflows and your interest in using the deep learning tools on Ampere based GPUs. As you have discovered, Ampere cards are not consistently supported by CUDA 10 based solutions. The deep learning collection includes eight separate GPU backed packages, which are all compiled against the CUDA Toolkit 10.1 release. Unfortunately, some of these dependencies have not been fully updated to support Ampere currently, and CUDA 10.1 does not natively support the binary formats Ampere provides.

Our plan for Pro 2.8 is to continue distributing a collection of installers and package set targeted at CUDA Tookilt 10.1, and works for GPUs of the last Kepler generation (3.7 with Tesla K80), Maxwell, Pascal, Turing, and Volta GPUs. This collection of packages has gone through extensive validation, and covers the GPUs for a broad base of our users, and has upstream support from the packages we are redistributing or building.

For Ampere users, we have been working toward a technology preview that will be available by UC 2021, which will have some known limitations but allow key arcgis.learn, Deep Learning based geoprocessing tools and core libraries to function, using CUDA 11 and cubins for the CC 8.6 platform. There won't be an installer for this technology preview, but it will be available as a conda installable metapackage, and once we post this, we will be sure to respond to this issue as well.

For the next release of Pro which will arrive Q4 2021, we will support Ampere out of the box as a target for the deep learning installers, and will have a solution based on CUDA Toolkit 11.

Thank you for all your comments and apologizes again for the delay in support.