nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Detect Objects Using Deep Learning Error with new RTX 3060

Hulseyj

Error appears when I use "Detect Objects Using Deep Learning" in ArcGIS Pro with new RTX3060 GPU. The model used is usa_building_footprints.dlpk. I am using ArcGIS Pro 2.7.3. I have no issues running the model with my GTX 1650 Super GPU, same computer, same settings. It appears ArcPro engages the GPU as the application is shown running on the GPU:

However, the program shows as "Running" but never progresses past 0%.

After about 30 minutes of "running", the following error message is returned:

I used the Input Parameters and Environment below. Again, it runs fine with the GTX 1650 Super GPU. All drivers are up to date on the new RTX3060.

Thank you for help or troubleshooting ideas.

Find more posts tagged with

Analysis

Accepted answers

All comments

Oliver_Morris

I am getting this error -

ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support (http://esriurl.com/support) to Report a Bug, and refer to the error help for potential solutions or workarounds.

Tile size should be greater than zero.
Failed to execute (DetectObjectsUsingDeepLearning).

Running on a 24GB GPU - NVIDIA A10G

Any idea where the issues lies. Full script:

with arcpy.EnvManager(mask="raster1", processorType="GPU"):
out_classified_raster = arcpy.ia.DetectObjectsUsingDeepLearning(
in_raster="Orthomosaic.rgb.tif",
out_detected_objects=r"C:\Users\Administrator\Documents\ArcGIS\Projects\MyProject\MyProject.gdb\raster1_sam_out3",
in_model_definition=r"C:\Users\Administrator\Downloads\SAM.dlpk",
arguments="padding 512;batch_size 8;box_nms_thresh 0.7;points_per_batch 128;stability_score_thresh 0.6;min_mask_region_area 0",
run_nms="NO_NMS",
confidence_score_field="Confidence",
class_value_field="Class",
max_overlap_ratio=0,
processing_mode="PROCESS_AS_MOSAICKED_IMAGE"
)
out_classified_raster.save(None)

Hulseyj

I am happy to report that Detect Objects Using Deep Learning is running successfully with my RTX 3060 after performing a clean install of ArcPro 2.9.0 and the 2.9 Deep Learning Libraries. I am seeing a significant improvement in speed as compared to my RTX 1650 Super. I have tested the Building Footprint Extraction deep learning model that is published by ESRI on the Living Atlas. I see better results in terms of speed when I use the default tile size environment settings (I remove the model argument: tile size 512). I have run the model in ArcPro and Jupyter notebook successfully.

Thanks to all who have worked to solve the issue with using the deep learning tools on the new Ampere cards.

borbalafarkas

Well, winter is coming ...

I tried ArcGIS Pro 2.9 a few weeks ago and conda install -c esri deep-learning-essentials, as there was no install pack yet. Did not work on RTX3080. Nor did ampere preview.

Today I installed 2.9 on another notebook and found that there's a new 2.9 installer for deep learning packages. This time the code (ESRI MaskRCNN training) is running in Jupyter notebook (had no time to test further or directly from ArcGIS yet, but I mainly use Jupyter anyway). BTW this time it is tensorflow 2.5.1 in the env.

I also see that in the esri-build channel there is a 3.0 pack in prep.

Thought you wanted to know ...

borbalafarkas

just FYI, (probably you already follow) this GitHub thread might be related:

Update for RTX 30 Series GPUs (CUDA 11) · Issue #17 · Esri/deep-learning-frameworks · GitHub

borbalafarkas

I also wonder whether there is any progress here.

The "preview" not just does not work but because if internal dependencies I was unable to fix it whatever I tried.

BTW, the preview is built with tensorflow 2.3.0 / tensorflow-gpu 2.3.0 (which is an upgrade from the 2.1.0 as was in the essentials pack). Isn't >=2.4.0 required (would be a better choice) for RTX 30 cards?

Anyway, I tried to upgrade to 2.4.0 and 2.5.0 (the ESRI conda page contains a 2.5.0 pack by ESRI, just a clue) but all failed. I mean, the updated 2.4.0 or 2.5.0 environments worked fine for other deep learning workflows (so the env is healthy otherwise) but do not work for the "standard" ESRI workflows (neither from Pro nor from Jupyter) as it was advertised to work. All my envs were cloned from arcgispro-py3 and contained the ESRI modules as much as dependencies permitted.

I used both RTX3050TI(4GB) and RTX3080(16GB) cards.

I also join to the above said:
- Is there anyone to confirm that the "preview" worked for them, please?

_Cartographer_

Any update on fixes for the Ampere preview? I don't think anyone has been able to get it working yet unfortunately.

Paul_Christensen

I am also seeing the same 002667 in Detect Objects...

I tried using the downloaded usa_building_footprints.dlpk and the online Living Atlas version.

Info and List reports appear identical to others.

RTX 3070

Side Note: Something to try for anyone experiencing verification errors when installing deep-learning-ampere-preview.

I could not get the deep-learning-ampere-preview to install because I kept getting tensorflow-base verification errors like "...package is corrupt, ... could not find file..." however I could manually explore to the file's location and see the file was there.

It turns out that several of their files are buried a dozen folders deep and the combination of my long username and the packages being stored in %localappdata%, the file path was over the Windows 10 limit of 260 characters. I modified the windows filepath limits to allow paths that exceed 260 characters but that did not seem to affect conda.

I had to create a new location for packages closer to root in order to decrease the file path length and it finally worked.

Amperecondainfo.txt

Amperecondalist.txt

SangeetMathew

Hi ,

If after following the steps from the github page do not fix the issue, Can you run the following commands and attach the Amperecondalist & Amperecondainfo files here. Thanks.

- In the python command type > Proenv

- Type conda list > c:\Amperecondalist.txt

-Type conda info > c:\Amperecondainfo.txt

SangeetMathew

This looks like an install issue. would like to point to the comment for the Esri DL github page for the steps to install incase something was missed . https://github.com/Esri/deep-learning-frameworks/issues/17#issuecomment-905895867

Also as mentioned on the page , this is just a tech preview , so there may be workflows that may not work.

Hulseyj

Below is the full error report when I ran the model from the Python Window. The screenshot above cut off some of the error read-out.

import arcpy
from arcpy.ia import *

# Check out the ArcGIS Image Analyst extension license
arcpy.CheckOutExtension("ImageAnalyst")

DetectObjectsUsingDeepLearning("c:/detect_objects/OP2020_449332_32.tif",
"c:/detect_objects/OP2020_449332_32.shp", "c:/detect_objects/usa_building_footprints.dlpk",
"padding 0; threshold 0.5; batch_size 4", "NO_NMS", "Confidence",
"Class", 0, "PROCESS_AS_MOSAICKED_IMAGE")
Traceback (most recent call last):
File "<string>", line 10, in <module>
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\ia\Functions.py", line 1805, in DetectObjectsUsingDeepLearning
processing_mode)
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\sa\Utils.py", line 53, in swapper
result = wrapper(*args, **kwargs)
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\ia\Functions.py", line 1794, in Wrapper
processing_mode)
File "C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\geoprocessing\_base.py", line 512, in <lambda>
return lambda *args: val(*gp_fixargs(args, True))
arcgisscripting.ExecuteError: Failed to execute. Parameters are not valid.
ERROR 002667: Unable to initialize python raster function with scalar arguments. [C:\Users\User\AppData\Local\Temp\ArcGISProTemp13908\usa_building_footprints.dlpk\ArcGISInstanceDetector.py]
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Temp\ArcGISProTemp13908\usa_building_footprints.dlpk\ArcGISInstanceDetector.py", line 142, in initialize
'{}.{}'.format(framework, self.json_info['ModelConfiguration'])), 'ChildInstanceDetector')
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\arcgis\__init__.py", line 4, in <module>
from . import (features, geoanalytics, geocoding, geometry)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\arcgis\features\__init__.py", line 28, in <module>
from . import enrich_data
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\arcgis\features\enrich_data.py", line 10, in <module>
import arcgis.network as network
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\arcgis\network\__init__.py", line 7, in <module>
from ._layer import NetworkLayer, NetworkDataset, ClosestFacilityLayer, ServiceAreaLayer, RouteLayer, NAJob, ODCostMatrixLayer
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\arcgis\network\_layer.py", line 8, in <module>
from arcgis.mapping import MapImageLayer
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\arcgis\mapping\__init__.py", line 8, in <module>
from ._types import WebMap, WebScene, MapImageLayer, MapImageLayerManager, VectorTileLayer, OfflineMapAreaManager, PackagingJob
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\arcgis\mapping\_types.py", line 25, in <module>
from arcgis.widgets._mapview._traitlets_extension import ObservableDict
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\arcgis\widgets\__init__.py", line 2, in <module>
from arcgis.widgets._mapview import MapView
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\arcgis\widgets\_mapview\__init__.py", line 1, in <module>
from arcgis.widgets._mapview._mapview import MapView
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\arcgis\widgets\_mapview\_mapview.py", line 37, in <module>
import pandas as pd
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\pandas\__init__.py", line 22, in <module>
from pandas.compat import (
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\pandas\compat\__init__.py", line 23, in <module>
from pandas.compat.pyarrow import (
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\pandas\compat\pyarrow.py", line 9, in <module>
_palv = Version(_pa_version)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\Pro28-Ampere\lib\site-packages\pandas\util\version\__init__.py", line 339, in __init__
match = self._regex.search(version)
TypeError: expected string or bytes-like object

Unable to initialize python raster function with scalar arguments.
Failed to execute (DetectObjectsUsingDeepLearning).

_Cartographer_

I am unfortunately getting the exact same error as well.

Hulseyj

Hi samaloysius4,

I successfully installed the new deep learning ampere preview in my new clone environment (thank you!). However, I am getting an error with the geoprocessing tool "Detect Objects Using Deep Learning', specifically with the Model Definition when I use the usa_building_footprints.dlpk. I tried a few other .dlpks from the Living Atlas and same error with the 3 or 4 that I tried. Please see the message I am getting associated with the Model Definition (screenshot attached). Any ideas? Screenshot 2021-08-25 221523.png

SangeetMathew

Hi,

A Tech preview package for those having Ampere cards has been uploaded. To use it please follow the following steps.

-Uninstall the existing Deep Learning installed packages. (you don't need to uninstall ArcGIS Pro 2.8)

-Install Pro 2.8.2

- Open python command prompt as admin

Run the following commands

conda create --clone arcgispro-py3 --name Pro28-Ampere --pinned

activate Pro28-Ampere

conda install -c esri deep-learning-ampere-preview

Proswap Pro28-Ampere

SangeetMathew

Thanks Paul.

Please uninstall the existing Deep Learning package first. Once the installer has been uninstalled , first make sure you have Pro 2.8.2 installed. Next it is advisable to create a clone of your default environment . You can do this with the following steps in your python command prompt

conda create --clone arcgispro-py3 --name clone --pinned

conda install -c esri deep-learning-ampere-preview

Paul_Christensen

deep-learning-ampere-preview is live on anaconda.org

https://anaconda.org/Esri/deep-learning-ampere-preview

also, cudatoolkit 11.2.0 is available

Cudatoolkit :: Anaconda.org

SangeetMathew

The tech preview is still being worked on . We will get back to you when we have it released . Thank you for your patience.

_Cartographer_

With UC 2021 underway, do we have any updates on the status of the technology preview for Ampere users?

SangeetMathew

Hi All,

Thank you for providing details on your workflows and your interest in using the deep learning tools on Ampere based GPUs. As you have discovered, Ampere cards are not consistently supported by CUDA 10 based solutions. The deep learning collection includes eight separate GPU backed packages, which are all compiled against the CUDA Toolkit 10.1 release. Unfortunately, some of these dependencies have not been fully updated to support Ampere currently, and CUDA 10.1 does not natively support the binary formats Ampere provides.

Our plan for Pro 2.8 is to continue distributing a collection of installers and package set targeted at CUDA Tookilt 10.1, and works for GPUs of the last Kepler generation (3.7 with Tesla K80), Maxwell, Pascal, Turing, and Volta GPUs. This collection of packages has gone through extensive validation, and covers the GPUs for a broad base of our users, and has upstream support from the packages we are redistributing or building.

For Ampere users, we have been working toward a technology preview that will be available by UC 2021, which will have some known limitations but allow key arcgis.learn, Deep Learning based geoprocessing tools and core libraries to function, using CUDA 11 and cubins for the CC 8.6 platform. There won't be an installer for this technology preview, but it will be available as a conda installable metapackage, and once we post this, we will be sure to respond to this issue as well.

For the next release of Pro which will arrive Q4 2021, we will support Ampere out of the box as a target for the deep learning installers, and will have a solution based on CUDA Toolkit 11.

Thank you for all your comments and apologizes again for the delay in support.

Anonymous User

Hey, so I was referred here from GitHub. Same story for me with RTX 3080. Using the Imagery Analyst DL toolset for me in ArcGIS Pro usually results in waiting 30 minutes for mysterious CUDA errors or false successes. Anyway, here is a list of issues I have encountered, some of which may not be directly related to GPU, but the "CUDA Assert" and "NaN" errors (resulting from training epochs) are big problems:

Just for the record, issues I've run into with my RTX 3080:
- Many "CUDA Assert" errors throughout my attempts to use tools in ArcGIS Pro.
- Create image chips tool refusing to process RCNN files for no apparent reason; PASCALs worked.
- False success with DL tools that claim to have been successful, but nothing whatsoever results from running the final model, even just to do a basic test for any result.
- Certain elements in ArcGIS Notebook not working at all, like show batch samples, which work in external Jupyter Notebook (though this could be unrelated to GPU).
- In external Jupyter Notebook/Training Model: Optimal Learning Rate graph feels buggy.
- In external Jupyter Notebook/Training Model: Training the model in any instance results in "NaN" values per epoch statistics, which is probably linked to previous point where ArcGIS DL Tools were "successful" but nothing meaningful was ever produced.

SangeetMathew

Thank you Nathan!

SangeetMathew

Thank you . We are working on it.

njpamperin

I’d be happy to test as well!

_Cartographer_

Any possible support before v2.9 would be seriously appreciated. That's a long time to wait -- and many more users will encounter this problem in the meantime. I have already volunteered to be an early adopter to help test if needed -- at least then I could use my GPU as originally intended. I'm sure a number of us would be happy to alpha/beta test if it meant getting this issue moving forward.

njpamperin

Thanks for the info and I appreciate the efforts to try to come up with a solution ahead of v2.9!

SangeetMathew

That is the timeframe for Pro v2.9 , but are looking into ways to possibly provide support for this earlier. Apologies for the inconvenience.

njpamperin

So that would be ~November for Pro v2.9?

SangeetMathew

Hi everyone ,

Thank you for your comments. Due to potential breaking changes, For ArcGIS Pro 2.8, the versions have not been updated. We are looking into upgrading the versions of CUDA , PyTorch, TensorFlow for the next release.

njpamperin

Getting the same error (below) trying to train deep learning models in some instances. I am running an Nvidia Geforce RTX 3080 (laptop). I can get the same models trained on a desktop Quadro RTX 5000 that has CUDA toolkit 11.2 installed but also has an instance of CUDA 10.1 installed as well. For what its worth...

File "C:\Users\natep\AppData\Local\ESRI\conda\envs\DeepLearning\Lib\site-packages\torchvision\ops\boxes.py", line 95, in remove_small_boxes
keep = keep.nonzero().squeeze(1)
RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered
Failed to execute (TrainDeepLearningModel).

_Cartographer_

This is 100% caused by lack of CUDA 11 support and lack of updated libraries for PyTorch, TensorFlow, etc.

The latest RTX 30 series GPUs require CUDA 11.

ESRI's current "Deep Learning Frameworks" as linked above use CUDA 10.1

Unfortunately, looking at the pre-release frameworks for ArcGIS Pro 2.8 (available here: https://anaconda.org/Esri/deep-learning-essentials/files) they have still not updated to CUDA 11, or updated PyTorch/TensorFlow. I do not understand why not. I think this comes down to demand.

Everyone who has this issue should be posting and making sure that ESRI knows this is a problem for an increasing number of users -- and will obviously only increase further as more people get the latest GPUs. With how much ESRI is touting ArcGIS deep learning capabilities, this slow response is quite surprising.

AndrewEastop

Have you tried downloading this?

Install Deep Learning Frameworks for the ArcGIS System

Hulseyj

Thanks Andrew. Glad to know I am not the only one experiencing the issue. No resolution yet.

Hulseyj

Thanks Tim. I added my name and issue to that post on GitHub.

AndrewEastop

Hi Hulseyj, I have just experienced the same problem as yourself. Did you find a resolution?

Tim_McGinnes

An issue was just posted to Esri's deep learning GitHub site about this (may be caused by a lack of CUDA 11 support in the underlying packages):

https://github.com/Esri/deep-learning-frameworks/issues/17

Probably worth adding your name to that as well - the more people raising it the more likely it might get fixed.

Hulseyj

Thanks, Dan. I have worked through the recommended steps in the blogs. The CUDA Toolkit has been installed, and running deviceQuery show both cards are CUDA-capable and detected.

BandwidthTest passed:

The Deep Learning Libraries for Arc Pro have been installed. I am using the default arcgispro-py3 environment.

I am not sure what else to try. The out-of-box model works fine with the GTX 1650 Super, but not with the new RTX 3060.

danretired

recent blogs on this and related topics especially since your hardware is new

Deep Learning with ArcGIS Pro Tips & Tricks: Part 1 (esri.com)

Deep Learning with ArcGIS Pro Tips & Tricks: Part 2 (esri.com)

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.