SSD model trained on colab got errors on ArcGIS Pro

1503
5
03-28-2021 11:17 PM
KyawNaingWin
New Contributor II

I am following a use-deep-learning-to-assess-palm-tree-health tutorial. Since I do not have GPU on my computer, I used colab to train the SSD model with arcgis.learn and it was successfully completed.

When I use the trained model (either .emd or dlpk) in "Detect Objects Using Deep Learning" tool, i got error right after model definition selection.

ERROR 002667 Unable to initialize python raster function with scalar arguments. [D:\ESRI_DL\palm_tree_health\ImageChips\models\palmtree_detector_ssd25\ArcGISObjectDetector.py]
Traceback (most recent call last):
File "C:\Users\Kyaw\AppData\Local\Temp\ArcGISProTemp21936\palmtree_detector_ssd25.dlpk\ArcGISObjectDetector.py", line 166, in initialize
self.child_object_detector.initialize(model, model_as_file)
File "C:\Users\Kyaw\AppData\Local\ESRI\conda\envs\arcgispro-py3-dl\Lib\site-packages\arcgis\learn\models\_inferencing\_DynamicSSD.py", line 195, in initialize
self.ssd = SingleShotDetector.from_emd(data=None, emd_path=model)
File "C:\Users\Kyaw\AppData\Local\ESRI\conda\envs\arcgispro-py3-dl\Lib\site-packages\arcgis\learn\models\_ssd.py", line 346, in from_emd
ssd = cls(data, emd['Grids'], emd['Zooms'], emd['Ratios'], pretrained_path=str(model_file), backend=backend, backbone=backbone, ssd_version=ssd_version)
File "C:\Users\Kyaw\AppData\Local\ESRI\conda\envs\arcgispro-py3-dl\Lib\site-packages\arcgis\learn\models\_ssd.py", line 228, in __init__
self.load(pretrained_path)
File "C:\Users\Kyaw\AppData\Local\ESRI\conda\envs\arcgispro-py3-dl\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 1300, in load
raise e
File "C:\Users\Kyaw\AppData\Local\ESRI\conda\envs\arcgispro-py3-dl\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 1298, in load
self.learn.load(name, purge=False)
File "C:\Users\Kyaw\AppData\Local\ESRI\conda\envs\arcgispro-py3-dl\Lib\site-packages\fastai\basic_train.py", line 269, in load
state = torch.load(source, map_location=device)
File "C:\Users\Kyaw\AppData\Local\ESRI\conda\envs\arcgispro-py3-dl\Lib\site-packages\torch\serialization.py", line 527, in load
with _open_zipfile_reader(f) as opened_zipfile:
File "C:\Users\Kyaw\AppData\Local\ESRI\conda\envs\arcgispro-py3-dl\Lib\site-packages\torch\serialization.py", line 224, in __init__
super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
AttributeError: 'WindowsPath' object has no attribute 'tell'

Unable to initialize python raster function with scalar arguments.

I tested with the pre-trained model provided on the tutorial page, it works without any problem.

What could be the problem? Has anybody train SSD on colab for ArcGIS Pro?

Tags (3)
0 Kudos
5 Replies
mvandoornik
New Contributor

I have the same error (also trained my model on Colab). In the past I fixed the error by upgrading my PyTorch package from 1.4.0 to 1.6.0. You could try this, I'm curious if that will work for you. With the pre-trained models from ESRI this upgrade is not necessary, for some reason.. I have no idea why. 

See also my post about this: Detect Objects Using Deep Learning: ERROR 002667 U... - Esri Community

0 Kudos
KyawNaingWin
New Contributor II

Thanks for sharing your experience.

I tried as you suggested but getting same error.

I set epochs =100 and use early_stopping to get the best model in all trainings. The model can detect palm trees on sub_set of image (about 1200 x 600) on the colab but miss a lot. It either did not work on ArcGIS Pro.

0 Kudos
mvandoornik
New Contributor

Last week I managed to fix the error again (I'm currently working on a new laptop with ArcGIS Pro 2.7). 

The error was solved after upgrading both Torch and Torchvision. It might be the Torchvison upgrade that fixed the error. I used the following command to get a stable CUDA-enabled version of both packages:

python -m pip install torch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

Hope this will help you. 

KyawNaingWin
New Contributor II

Not succeeded yet. Will try again.

Yesterday, I trained the model on AGOL notebook where I can get a pre installed run time with GPU costing 0.5 credit per min.

Used the same codes I had tried on the Colab. All the modules needed for ArcGIS deep learning training are already installed there. The model trained on AGOL notebook work on my ArcGIS Pro. Currently AGOL notebook runtime use Arcgis 1.8.1 module. I have retrieved all the module names of the runtime and will install them on the Colab and test again.

Best

0 Kudos
DanPatterson
MVP Esteemed Contributor
0 Kudos