topic Re: Train Deep learning Model. in Imagery and Remote Sensing Questions

Train Deep learning Model.

epiewesner — Fri, 16 Sep 2022 10:51:48 GMT

Greetings. I am trying to train my data using the deep learning model, but i keep getting this error message. Can someone help me out.

Traceback (most recent call last):
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 308, in execute
data_bunch = prepare_data(in_folders, working_dir=out_folder, **prepare_data_kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\_data.py", line 1440, in prepare_data
raise Exception(
Exception: Could not infer dataset type. Please specify a supported dataset type or ensure that the path contains valid esri files

During handling of the above exception, another exception occurred:

Failed script (null)...
Failed to execute (TrainDeepLearningModel).

Re: Train Deep learning Model.

PavanYadav — Fri, 16 Sep 2022 15:57:23 GMT

Hello @epiewesner

In the above error, I see this: "Could not infer dataset type. Please specify a supported dataset type or ensure that the path contains valid esri files"

Looks like the Training Deep Learning Model tool is finding some issues with the data that you are trying to use. A few things:

1. Are you using ArcGIS Pro or other application maybe one of Notebook examples outside of ArcGIS Pro? If using a notebook and if it's one of Esri's examples, can you please share the link? You might want to see this if using notebook example - https://community.esri.com/t5/arcgis-pro-questions/model-training-failure-in-arcgis-pro/td-p/1031634

2. If using ArcGIS Pro, what version are you currently on? at what version the data was exported (assuming you use the Export Training Data tool)?

3. Any changes made to the .json or any other files of the data?

4. Is the language and regional settings are same between now and when the data was exported?

Thanks

Pavan

Re: Train Deep learning Model.

epiewesner — Fri, 16 Sep 2022 17:46:32 GMT

Hi @PavanYadav, thank you for the help. But i got it sorted, unfortunately, i have ran into another issue. Hopping you can help me. Below is the new error message i am getting.

Traceback (most recent call last):
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 390, in <module>
execute()
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 334, in execute
training_model_object.fit(
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 902, in fit
lr = self.lr_find(allow_plot=False)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 721, in lr_find
raise e
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 718, in lr_find
self.learn.lr_find()
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\train.py", line 40, in lr_find
epochs = int(np.ceil(num_it/len(learn.data.train_dl))) * (num_distrib() or 1)
ZeroDivisionError: division by zero

Re: Train Deep learning Model.

PavanYadav — Thu, 20 Oct 2022 20:09:09 GMT

Glad to hear that you fixed the first issue? Can ask how you fixed it?
For the second issue you can look at the the stats.txt file. Please see if you have any classes that don't have samples.

Re: Train Deep learning Model.

epiewesner — Fri, 16 Sep 2022 19:25:51 GMT

@PavanYadav the issues was with my training samples. I was adding only the image_chips. But when i added all the data it started working.

Re: Train Deep learning Model.

KevinRathgeber1 — Mon, 14 Nov 2022 17:56:58 GMT

@epiewesner What do you mean by "all the data". I am having same division by zero error - I have added the folder that contains the images and labels and the stats.txt, maps.txt, esri_accumulated_stats.json and exri_model_definition.emd files. Can't seem to figure out how to get around that error.

Re: Train Deep learning Model.

PavanYadav — Mon, 14 Nov 2022 18:09:44 GMT

@KevinRathgeber1 right above your post, please see @epiewesner's response.
The Export Training Data for Deep Learning tool typically outputs a folder containing something like the following

In this case, the input to the Train Deep Learning Model tool is: "FormatTest_PNG_KITTI_rectangles" folder.

Re: Train Deep learning Model.

KevinRathgeber1 — Mon, 14 Nov 2022 22:27:41 GMT

My bad, you are correct. I reread the above and realised they were getting the Divide by zero error after they pointed to the correct folder. That is the problem I am having. I have pointed to the folder as you have pointed out above, then when I run it I get the exact same errors.

Traceback (most recent call last):
File "c:\program files\arcgis\pro\Resources\ArcToolbox\Toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 390, in <module>
execute()
File "c:\program files\arcgis\pro\Resources\ArcToolbox\Toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 334, in execute
training_model_object.fit(
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 902, in fit
lr = self.lr_find(allow_plot=False)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 721, in lr_find
raise e
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 718, in lr_find
self.learn.lr_find()
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\train.py", line 40, in lr_find
epochs = int(np.ceil(num_it/len(learn.data.train_dl))) * (num_distrib() or 1)
ZeroDivisionError: division by zero

Failed script (null)...
Failed to execute (TrainDeepLearningModel).

Re: Train Deep learning Model.

PavanYadav — Tue, 15 Nov 2022 19:32:04 GMT

@KevinRathgeber1 Can you share how many training samples you have and what number are you using in the Validation %.?

Here is an example - let's say I have 4 samples and I use 10 in the % validation parameter.. so in this case numbers of samples for validation will be 0.4, which is not possible it should be a whole number.

Re: Train Deep learning Model.

KevinRathgeber1 — Tue, 15 Nov 2022 23:19:00 GMT

Yeah that was the problem I had 3 samples and 10%. I added one more sample for 4 and then did two runs set to 50% and 75% and each of those worked.

Re: Train Deep learning Model.

PavanYadav — Wed, 16 Nov 2022 00:21:01 GMT

@KevinRathgeber1 glad you have fixed the problem. Was it for a test? Because 4 samples are very little.

I often hear that we need at least 1000 samples for each category; but it really depends on many things for example some architectures needs no or very minimum labels. DetReg supported in ArcGIS is one example that needs much less labels comparing to other model architectures. You might find these interesting: https://sites.uab.edu/periop-datascience/2021/06/28/sample-size-in-machine-learning-and-artificial-intelligence/#:~:text=If%20you've%20talked%20with,least%201%2C000%20samples%20per%20class , https://machinelearningmastery.com/much-training-data-required-machine-learning/ and https://towardsdatascience.com/how-do-you-know-you-have-enough-training-data-ad9b1fd679ee

If you want to try out an end-to-end deep learning workflow in ArcGIS, these are good tutorials:
https://learn.arcgis.com/en/projects/use-deep-learning-to-assess-palm-tree-health/
https://learn.arcgis.com/en/projects/automate-fire-damage-assessment-with-deep-learning/

Re: Train Deep learning Model.

KevinRathgeber1 — Wed, 16 Nov 2022 15:14:03 GMT

Yeah I was just playing around with the concept of it trying to get it working. And from the results I got I could tell it was very little lol. Thanks for your help.

Re: Train Deep learning Model.

PawanThapa2 — Mon, 17 Jun 2024 18:02:42 GMT

I have a small area, so I change image chip 128 instead of 512 to extract training sample, it works.

If I use 512, then I get only three samples so I decrease it to 128

Also, I faced other error: Error:Error(s) in loading state_dict for _DeepLabOverride: size mismatch for backbone.conv1.weight: copying a param with shape torch.Size([64, 3, 7, 7]) from checkpoint, the shape in current model is torch.Size([64, 4, 7, 7])..Training was not sucessful

then use extract band make three bands, it solved

I have 7 samples and I use 0.5 in the % validation parameter, to make whole number. Still I got the Error:division by zero.Training was not successful.