Error: Could not find 'NumPixelsPerClass' in step Train Deep Learning Model

NayaraV

Hi, I received this error when I tried to Train Deep Learning Model. (Arcgis Pro 2.7)
I created the label with "Label Objects for Deep Learning" with 1 class. I exported the training with metadata as "Classified Tiles".
My Error Message:
Could not find 'NumPixelsPerClass' in 'esri_accumulated_stats.json'. Ignoring `class_balancing` parameter.
Traceback (most recent call last):
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 232, in <module>
execute()
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 207, in execute
show_accuracy=show_accuracy)]
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\deeplearning\lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 708, in fit
self.learn.fit_one_cycle(epochs, lr, callbacks=callbacks, **kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\deeplearning\lib\site-packages\fastai\train.py", line 23, in fit_one_cycle
learn.fit(cyc_len, max_lr, wd=wd, callbacks=callbacks)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\deeplearning\lib\site-packages\fastai\basic_train.py", line 200, in fit
fit(epochs, self, metrics=self.metrics, callbacks=self.callbacks+callbacks)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\deeplearning\lib\site-packages\fastai\basic_train.py", line 101, in fit
loss = loss_batch(learn.model, xb, yb, learn.loss_func, learn.opt, cb_handler)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\deeplearning\lib\site-packages\fastai\basic_train.py", line 34, in loss_batch
if not skip_bwd: loss.backward()
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\deeplearning\lib\site-packages\torch\tensor.py", line 195, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\deeplearning\lib\site-packages\torch\autograd\__init__.py", line 99, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 4.00 GiB total capacity; 2.72 GiB already allocated; 60.95 MiB free; 2.89 GiB reserved in total by PyTorch)
Ocorreu um erro ao executar (TrainDeepLearningModel)

Please, anyone could help me? I've been working for days trying to fix it
Kindly
Nayara

DrVSSKiran

Hi,

Try to reduce the tile size during the exporting and batch size during the training. 

For example:

If you are working on tile size 448 as mentioned in below figure try to reduce it 50%

Secondly, at the time of training the dataset please reduce the batch size is 1 if you worked on 2 or 4 as mentioned the below figure.

If still, you will get the same error, kindly check the GPU usage and assign complete GPU for ArcGIS Pro. and Check the python libraries, uninstalled the fast.ai and install the fast.ai and try.

Good Luck.

Thanks