nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Train Deep learning Model.

epiewesner

Groeten. Ik probeer mijn gegevens te trainen met het deep learning model, maar ik blijf deze foutmelding krijgen. Kan iemand mij helpen.<\/P>

Traceback (meest recente oproep laatst):
Bestand "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", regel 308, in execute
data_bunch = prepare_data(in_folders, working_dir=out_folder, **prepare_data_kwargs)
Bestand "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\_data.py", regel 1440, in prepare_data
raise Exception(
Exception: Kon het datasettype niet afleiden. Specificeer een ondersteund datasettype of zorg ervoor dat het pad geldige esri-bestanden bevat.<\/P>

Tijdens het afhandelen van de bovenstaande uitzondering, deed zich een andere uitzondering voor:<\/P>

Traceback (meest recente oproep laatst):
Bestand "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", regel 390, in <module>
execute()
Bestand "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", regel 384, in execute
del data_bunch
UnboundLocalError: lokale variabele 'data_bunch' verwezen voordat deze was toegewezen.<\/P>

Mislukt script (null)...
Uitvoeren mislukt (TrainDeepLearningModel).<\/P>

Vind meer berichten getagd met

Analysis

Desktop

Accepted answers

epiewesner

@PavanYadav the issues was with my training samples. I was adding only the image_chips. But when i added all the data it started working.

PavanYadav

Glad to hear that you fixed the first issue? Can ask how you fixed it?
For the second issue you can look at the the stats.txt file. Please see if you have any classes that don't have samples.

All comments

PawanThapa2

I have a small area, so I change image chip 128 instead of 512 to extract training sample, it works.

If I use 512, then I get only three samples so I decrease it to 128

Also, I faced other error: Error:Error(s) in loading state_dict for _DeepLabOverride: size mismatch for backbone.conv1.weight: copying a param with shape torch.Size([64, 3, 7, 7]) from checkpoint, the shape in current model is torch.Size([64, 4, 7, 7])..Training was not sucessful

then use extract band make three bands, it solved

I have 7 samples and I use 0.5 in the % validation parameter, to make whole number. Still I got the Error:division by zero.Training was not successful.

KevinRathgeber1

Yeah I was just playing around with the concept of it trying to get it working. And from the results I got I could tell it was very little lol. Thanks for your help.

PavanYadav

@KevinRathgeber1 glad you have fixed the problem. Was it for a test? Because 4 samples are very little.

I often hear that we need at least 1000 samples for each category; but it really depends on many things for example some architectures needs no or very minimum labels. DetReg supported in ArcGIS is one example that needs much less labels comparing to other model architectures. You might find these interesting: https://sites.uab.edu/periop-datascience/2021/06/28/sample-size-in-machine-learning-and-artificial-intelligence/#:~:text=If%20you've%20talked%20with,least%201%2C000%20samples%20per%20class , https://machinelearningmastery.com/much-training-data-required-machine-learning/ and https://towardsdatascience.com/how-do-you-know-you-have-enough-training-data-ad9b1fd679ee

If you want to try out an end-to-end deep learning workflow in ArcGIS, these are good tutorials:
https://learn.arcgis.com/en/projects/use-deep-learning-to-assess-palm-tree-health/
https://learn.arcgis.com/en/projects/automate-fire-damage-assessment-with-deep-learning/

KevinRathgeber1

Yeah that was the problem I had 3 samples and 10%. I added one more sample for 4 and then did two runs set to 50% and 75% and each of those worked.

PavanYadav

@KevinRathgeber1 Can you share how many training samples you have and what number are you using in the Validation %.?

Here is an example - let's say I have 4 samples and I use 10 in the % validation parameter.. so in this case numbers of samples for validation will be 0.4, which is not possible it should be a whole number.

KevinRathgeber1

My bad, you are correct. I reread the above and realised they were getting the Divide by zero error after they pointed to the correct folder. That is the problem I am having. I have pointed to the folder as you have pointed out above, then when I run it I get the exact same errors.

Traceback (most recent call last):
File "c:\program files\arcgis\pro\Resources\ArcToolbox\Toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 390, in <module>
execute()
File "c:\program files\arcgis\pro\Resources\ArcToolbox\Toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 334, in execute
training_model_object.fit(
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 902, in fit
lr = self.lr_find(allow_plot=False)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 721, in lr_find
raise e
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 718, in lr_find
self.learn.lr_find()
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\train.py", line 40, in lr_find
epochs = int(np.ceil(num_it/len(learn.data.train_dl))) * (num_distrib() or 1)
ZeroDivisionError: division by zero

Failed script (null)...
Failed to execute (TrainDeepLearningModel).

PavanYadav

@KevinRathgeber1 right above your post, please see @epiewesner's response.
The Export Training Data for Deep Learning tool typically outputs a folder containing something like the following

In this case, the input to the Train Deep Learning Model tool is: "FormatTest_PNG_KITTI_rectangles" folder.

KevinRathgeber1

@epiewesner What do you mean by "all the data". I am having same division by zero error - I have added the folder that contains the images and labels and the stats.txt, maps.txt, esri_accumulated_stats.json and exri_model_definition.emd files. Can't seem to figure out how to get around that error.

epiewesner

@PavanYadav the issues was with my training samples. I was adding only the image_chips. But when i added all the data it started working.

PavanYadav

Glad to hear that you fixed the first issue? Can ask how you fixed it?
For the second issue you can look at the the stats.txt file. Please see if you have any classes that don't have samples.

epiewesner

Hi @PavanYadav, thank you for the help. But i got it sorted, unfortunately, i have ran into another issue. Hopping you can help me. Below is the new error message i am getting.

Traceback (most recent call last):
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 390, in <module>
execute()
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 334, in execute
training_model_object.fit(
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 902, in fit
lr = self.lr_find(allow_plot=False)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 721, in lr_find
raise e
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\arcgis\learn\models\_arcgis_model.py", line 718, in lr_find
self.learn.lr_find()
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\fastai\train.py", line 40, in lr_find
epochs = int(np.ceil(num_it/len(learn.data.train_dl))) * (num_distrib() or 1)
ZeroDivisionError: division by zero

PavanYadav

Hello @epiewesner

In the above error, I see this: "Could not infer dataset type. Please specify a supported dataset type or ensure that the path contains valid esri files"

Looks like the Training Deep Learning Model tool is finding some issues with the data that you are trying to use. A few things:

1. Are you using ArcGIS Pro or other application maybe one of Notebook examples outside of ArcGIS Pro? If using a notebook and if it's one of Esri's examples, can you please share the link? You might want to see this if using notebook example - https://community.esri.com/t5/arcgis-pro-questions/model-training-failure-in-arcgis-pro/td-p/1031634

2. If using ArcGIS Pro, what version are you currently on? at what version the data was exported (assuming you use the Export Training Data tool)?

3. Any changes made to the .json or any other files of the data?

4. Is the language and regional settings are same between now and when the data was exported?

Thanks

Pavan

Doe mee aan het gesprek

Aangemelde leden kunnen berichten plaatsen, updates volgen en meer. Nieuw hier? Registreer een gratis account.

Snelle Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.