Select to view content in your preferred language

deep learning error

1182
4
09-08-2022 07:06 PM
GengshuoHu
Emerging Contributor

I am using the deep learning product to do the landuse and there is a problem.

Here is the error message:

Classify Pixels Using Deep Learning
=====================
Parameters

Input Raster Surface Reflectance_LC08_L2SP_089079_20130427_20200912_02_T1_MTL
Output Classified Raster C:\Users\Admin\Documents\ArcGIS\Projects\MyProject1\MyProject1.gdb\SurfaceReflect_ClassifyPixel3
Model Definition C:\学习文件\2022 Semester 2\Final Research\土地利用深度学习\LandCoverClassification.dlpk
Arguments padding 128;batch_size 4;predict_background True;test_time_augmentation True;tile_size 512;landsat_imagery_level 2
Processing Mode PROCESS_AS_MOSAICKED_IMAGE
Output Folder
=====================
Environments

Extent 152.607318628634 -27.5730888984295 153.480255433056 -27.3123415412645 GEOGCS["GCS_GDA_1994",DATUM["D_GDA_1994",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]]
Cell Size 30
Processor Type GPU
=====================
Messages

Start Time: Friday, 9 September 2022 2:23:47 AM
ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support (http://esriurl.com/support) to Report a Bug, and refer to the error help for potential solutions or workarounds.
Unable to read pixels from the python raster function.
Unable to read pixels from the python raster function.
Function Read Error [Raster Function Template]
Unable to read pixels from the python raster function.
Traceback (most recent call last):
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\ArcGISImageClassifier.py", line 275, in updatePixels
xx = self.child_image_classifier.updatePixelsTTA(tlc, shape, props, **pixelBlocks).astype(props['pixelType'], copy=False)
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\_unet.py", line 443, in updatePixelsTTA
all_activations = self.tta_predict(
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\_unet.py", line 399, in tta_predict
int_surface = self.split_predict_interpolate(
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\_unet.py", line 371, in split_predict_interpolate
output = self.model(patches)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\layers.py", line 136, in forward
nres = l(res)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\layers.py", line 136, in forward
nres = l(res)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\layers.py", line 150, in forward
def forward(self, x): return torch.cat([x,x.orig], dim=1) if self.dense else (x+x.orig)
RuntimeError: CUDA out of memory. Tried to allocate 412.00 MiB (GPU 0; 2.00 GiB total capacity; 1.11 GiB already allocated; 0 bytes free; 1.41 GiB reserved in total by PyTorch)
Python raster function's .updatePixels() method returned nothing.
Unable to read pixels from the python raster function.
Unable to read pixels from the python raster function.
Function Read Error [Raster Function Template]
Unable to read pixels from the python raster function.
Traceback (most recent call last):
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\ArcGISImageClassifier.py", line 275, in updatePixels
xx = self.child_image_classifier.updatePixelsTTA(tlc, shape, props, **pixelBlocks).astype(props['pixelType'], copy=False)
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\_unet.py", line 443, in updatePixelsTTA
all_activations = self.tta_predict(
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\_unet.py", line 399, in tta_predict
int_surface = self.split_predict_interpolate(
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\_unet.py", line 371, in split_predict_interpolate
output = self.model(patches)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\layers.py", line 136, in forward
nres = l(res)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\layers.py", line 136, in forward
nres = l(res)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\layers.py", line 150, in forward
def forward(self, x): return torch.cat([x,x.orig], dim=1) if self.dense else (x+x.orig)
RuntimeError: CUDA out of memory. Tried to allocate 412.00 MiB (GPU 0; 2.00 GiB total capacity; 1.11 GiB already allocated; 0 bytes free; 1.41 GiB reserved in total by PyTorch)
Python raster function's .updatePixels() method returned nothing.
Unable to read pixels from the python raster function.
Unable to read pixels from the python raster function.
Function Read Error [Raster Function Template]
Unable to read pixels from the python raster function.
Traceback (most recent call last):
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\ArcGISImageClassifier.py", line 275, in updatePixels
xx = self.child_image_classifier.updatePixelsTTA(tlc, shape, props, **pixelBlocks).astype(props['pixelType'], copy=False)
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\_unet.py", line 443, in updatePixelsTTA
all_activations = self.tta_predict(
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\_unet.py", line 399, in tta_predict
int_surface = self.split_predict_interpolate(
File "C:\Users\Admin\AppData\Local\Temp\ArcGISProTemp14780\LandCoverClassification.dlpk\_unet.py", line 371, in split_predict_interpolate
output = self.model(patches)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\layers.py", line 136, in forward
nres = l(res)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\layers.py", line 136, in forward
nres = l(res)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\layers.py", line 150, in forward
def forward(self, x): return torch.cat([x,x.orig], dim=1) if self.dense else (x+x.orig)
RuntimeError: CUDA out of memory. Tried to allocate 412.00 MiB (GPU 0; 2.00 GiB total capacity; 1.11 GiB already allocated; 0 bytes free; 1.41 GiB reserved in total by PyTorch)
Python raster function's .updatePixels() method returned nothing.
Failed to execute (ClassifyPixelsUsingDeepLearning).
Failed at Friday, 9 September 2022 2:25:08 AM (Elapsed Time: 1 minutes 20 seconds)

0 Kudos
4 Replies
DanPatterson
MVP Esteemed Contributor

from

Deep learning frequently asked questions—ArcGIS Pro | Documentation

seeing a lot of  CUDA out of memory

errors in the error message.  do you have the recommended gpu memory? and have you experimented with the batch size?


... sort of retired...
0 Kudos
GengshuoHu
Emerging Contributor

Thanks a lot. It is time to change a new computer though.

0 Kudos
ShaunWalbridge
Esri Regular Contributor

The current system requirements for Pro 3.0 GPGPU workflows is for 6GB+ of GPU memory, and in the past this was set at a minimum of 4GB. We unfortunately can't support smaller memory cards because of the size of the underlying models we are building. That said, producing more useful errors in this case is something we intend on doing in future releases, and there are other GPGPU workflows like the SA tools that will work with less memory.

0 Kudos
GengshuoHu
Emerging Contributor

Thanks a lot!!

0 Kudos