Detect Object Using Deep Learning Error 999999 When GPU in used

bhariadi — Sat, 14 Aug 2021 07:50:03 GMT

Hello All,

I successfully detect object using deep learning, processed using CPU as followed pictured. 😊

However, when I use GPU to run this tool, the process end with error. 🤔

Those are my laptop specification:

DELL Precision 5550
Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz 2.40 GHz
32.0 GB of RAM
Windows 10 Pro
NVIDIA Quadro T2000 with Max-Q Design
ArcGIS Pro 2.8.2

I installed ArcGIS Pro Deep Learning Framework, and get the installer from this site.

This is the error messages, I copied from ArcGIS Pro:

Detect Objects Using Deep Learning
=====================
Parameters

Input Raster 47_res10cm_micasensemx.tif
Output Detected Objects C:\Users\Administrator\Documents\ArcGIS\Projects\20210813_ArcGISPro_BelajarDeepLearning_1\20210813_ArcGISPro_BelajarDeepLearning_1.gdb\c47_res10cm_micasensemx_DetectObject1_GPU
Model Definition C:\DeepLearning\models\OPDetectionModel_Res10cm_YOLOv3_v11_Stage1_scorexxxx.dlpk
Arguments padding 56;threshold 0,1;nms_overlap 0,5;batch_size 4;exclude_pad_detections True
Non Maximum Suppression NMS
Confidence Score Field Confidence
Class Value Field Class
Max Overlap Ratio 0,5
Processing Mode PROCESS_AS_MOSAICKED_IMAGE
=====================
Environments

Scratch Workspace C:\Users\Administrator\Documents\ArcGIS\Projects\20210813_ArcGISPro_BelajarDeepLearning_1\20210813_ArcGISPro_BelajarDeepLearning_1.gdb
Extent 288571,076325618 9786953,0828514 288891,75196697 9787250,26344576
Processor Type GPU
Current Workspace C:\Users\Administrator\Documents\ArcGIS\Projects\20210813_ArcGISPro_BelajarDeepLearning_1\20210813_ArcGISPro_BelajarDeepLearning_1.gdb
=====================
Messages

Start Time: 13 August 2021 17:31:38
ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support (http://esriurl.com/support) to Report a Bug, and refer to the error help for potential solutions or workarounds.
Python raster function is unable to vectorize the data. [Failed to generate table]
Python raster function is unable to vectorize the data.
Traceback (most recent call last):
File "C:\Users\ADMINI~1\AppData\Local\Temp\ArcGISProTemp12504\OPDetectionModel_Res10cm_YOLOv3_v11_Stage1_scorexxxx.dlpk\ArcGISObjectDetector.py", line 215, in vectorize
polygon_list, scores, classes = self.child_object_detector.vectorize(**pixelBlocks)
File "C:\Users\Administrator\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\deeplearning282\Lib\site-packages\arcgis\learn\models\_inferencing\_yolov3_inference.py", line 334, in vectorize
for feature_idx in range(len(image_bbox.data[0])):
File "C:\Users\Administrator\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\deeplearning282\Lib\site-packages\fastai\vision\image.py", line 378, in data
lbls = np.array([o.data for o in lbls]) if lbls is not None else None
File "C:\Users\Administrator\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\deeplearning282\Lib\site-packages\torch\tensor.py", line 486, in __array__
return self.numpy()
TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
Python Raster Function's .vectorize() method returned nothing.
Python raster function is unable to vectorize the data. [Failed to generate table]
Python raster function is unable to vectorize the data.
Traceback (most recent call last):
File "C:\Users\ADMINI~1\AppData\Local\Temp\ArcGISProTemp12504\OPDetectionModel_Res10cm_YOLOv3_v11_Stage1_scorexxxx.dlpk\ArcGISObjectDetector.py", line 215, in vectorize
polygon_list, scores, classes = self.child_object_detector.vectorize(**pixelBlocks)
File "C:\Users\Administrator\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\deeplearning282\Lib\site-packages\arcgis\learn\models\_inferencing\_yolov3_inference.py", line 334, in vectorize
for feature_idx in range(len(image_bbox.data[0])):
File "C:\Users\Administrator\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\deeplearning282\Lib\site-packages\fastai\vision\image.py", line 378, in data
lbls = np.array([o.data for o in lbls]) if lbls is not None else None
File "C:\Users\Administrator\AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\deeplearning282\Lib\site-packages\torch\tensor.py", line 486, in __array__
return self.numpy()
TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
Python Raster Function's .vectorize() method returned nothing.
Failed to execute (DetectObjectsUsingDeepLearning).
Failed at 13 August 2021 17:32:01 (Elapsed Time: 23,14 seconds)

Image below is the information from nvidia-smi:

Hopefully anybody here can help me. Thank you.

Re: Detect Object Using Deep Learning Error 999999 When GPU in used

bhariadi — Sat, 14 Aug 2021 08:50:33 GMT

Update information (but it is not solving my previous problem yet):

I tried to train my data trough Jupyter Notebook & the CUDA worked. But when I used Detect Object Using Deep Learning tool in ArcGIS Pro, the error still persistent.

See the picture below.

Re: Detect Object Using Deep Learning Error 999999 When GPU in used

bhariadi — Thu, 18 Nov 2021 09:36:53 GMT

The problem solved.

Only need to update the ArcGIS Pro version to 2.9.

Cuda enabled

Re: Detect Object Using Deep Learning Error 999999 When GPU in used

shlirkd — Tue, 21 Dec 2021 10:11:24 GMT

Hello bhariadi,

I am not able to get the yolov3 model working using the detect objects using deep learning method.

How did you do that?

Regards

Re: Detect Object Using Deep Learning Error 999999 When GPU in used

bhariadi — Wed, 22 Dec 2021 04:17:14 GMT

Hi,

Usually. I am using Jupyter Notebook not via ArcGIS Pro.

Regards,

Re: Detect Object Using Deep Learning Error 999999 When GPU in used

Kris_AB — Thu, 06 Apr 2023 09:34:02 GMT

I have similar issue. error message while using detect object using deep learning in Arcgis pro. I used pretrained building footprint model. any suggestion?

topic Re: Detect Object Using Deep Learning Error 999999 When GPU in used in ArcGIS Pro Questions

Detect Object Using Deep Learning Error 999999 When GPU in used

Re: Detect Object Using Deep Learning Error 999999 When GPU in used

Re: Detect Object Using Deep Learning Error 999999 When GPU in used

Re: Detect Object Using Deep Learning Error 999999 When GPU in used

Re: Detect Object Using Deep Learning Error 999999 When GPU in used

Re: Detect Object Using Deep Learning Error 999999 When GPU in used