topic ERROR WHEN TRAINING DEEP MODEL in ArcGIS Image Analyst Questions

ERROR WHEN TRAINING DEEP MODEL

AthuleSali — Thu, 22 Sep 2022 01:04:14 GMT

Hello everyone...

I'm new in arcgis pro and am trying to create a deep learning model so now im stuck in training the model and im getting this error:

Traceback (most recent call last):
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 308, in execute
data_bunch = prepare_data(in_folders, working_dir=out_folder, **prepare_data_kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3-clone-Athu_DL\Lib\site-packages\arcgis\learn\_data.py", line 1440, in prepare_data
raise Exception(
Exception: Could not infer dataset type. Please specify a supported dataset type or ensure that the path contains valid esri files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 390, in <module>
execute()
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 384, in execute
del data_bunch
UnboundLocalError: local variable 'data_bunch' referenced before assignment

Failed script (null)...
Failed to execute (TrainDeepLearningModel).

Re: ERROR WHEN TRAINING DEEP MODEL

DanPatterson — Thu, 22 Sep 2022 01:25:49 GMT

Which of the required inputs are you using?

Train Deep Learning Model (Image Analyst)—ArcGIS Pro | Documentation

Re: ERROR WHEN TRAINING DEEP MODEL

AthuleSali — Thu, 22 Sep 2022 06:27:34 GMT

I used the results folder that I got from exported training data for deep learning.

Re: ERROR WHEN TRAINING DEEP MODEL

PavanYadav — Fri, 23 Sep 2022 18:57:23 GMT

hi @AthuleSali

This error indicates the issue is with the input data: Exception: Could not infer dataset type. Please specify a supported dataset type or ensure that the path contains valid esri files

Can you please share screen captures the following?
1. input folder contents

2. stats.txt

3. The Train Deep Learning Tool with parameters filled (or, even better if you can share the command. You can copy the python command from the history)?

Thanks

Pavan

Re: ERROR WHEN TRAINING DEEP MODEL

AthuleSali — Mon, 03 Oct 2022 16:08:06 GMT

Hellow @PavanYadav

Thanks for trying to assist, my mistake was really on the input data but I fixed it thank you so much.

Re: ERROR WHEN TRAINING DEEP MODEL

AthuleSali — Mon, 03 Oct 2022 16:27:57 GMT

@PavanYadav can you assist things I can do to get the accurate model? I have have been developing the Unet models since last week and my F1, Precision and Recall are so low. See the attached.

Re: ERROR WHEN TRAINING DEEP MODEL

PavanYadav — Fri, 07 Oct 2022 18:03:17 GMT

Hello @AthuleSali
Not sure if you have already considered the following but there are some general considerations to improve a deep learning model:

1. Utilize the max potential of your training data by using a max possible numbers of epochs (e.g. 100 or 500) and use the STOP_TRAINING—Early parameter. The model training will stop when the model is no longer improving, regardless of the max_epochs parameter value specified. This is the default. Let's say you entered 500 epochs but the tool does not find the loss decreasing after 75th epoch so it will stop. Now, here if you are still not satisfied with the quality, please consider following.

2. Improve the model by adding more training data in terms of both variety and number of samples.

3. You can consider changing the training and validation ratio, by default the tool uses 10% of data for validation. You may consider making this to 20%, in case the model is overfitting on the training data.

4. Each classes of sample should have enough samples.

The Train Deep Learning tool does have addition functionalities like data augmentation (randomly rotating, zooming) to create variety. There are other hyperparameters that you can experiment like Learning rate, but I will first try out the above.

Re: ERROR WHEN TRAINING DEEP MODEL

JoabelBarbieri — Thu, 02 Nov 2023 14:16:47 GMT

Getting these errors left and right. It seems the deep learning tools in Pro weren't refined and debugged enough before being released. It's disappointing.

Re: ERROR WHEN TRAINING DEEP MODEL

AlexaVerónicaGarcíaBarrantes — Sat, 04 Nov 2023 16:36:31 GMT

Hello AthuleSali

Im having the same issue, what did you do to fix it?

Re: ERROR WHEN TRAINING DEEP MODEL

GBacon — Wed, 10 Apr 2024 12:32:45 GMT

I'm having the same issue in Pro v3.2.2. There are tiffs in my \images folder and xml files in my \undefined\labels folder. I've tried running the tool with/without the labels folder. Without the labels folder, I get this error, making me think it needs the labels folder.

However, when I try to include my labels folder, I get the error below even though the exported training files used RCNN masks for the metadata format.

@PavanYadav I appreciate any assistance you can offer.

Re: ERROR WHEN TRAINING DEEP MODEL

PavanYadav — Fri, 12 Apr 2024 20:30:18 GMT

hello @GBacon sorry that I missed to see your message.

In the Train Deep Learning Model tool, you need to input the entire folder that is created by the Export Training Data For Deep Learning tool

Re: ERROR WHEN TRAINING DEEP MODEL

GBacon — Mon, 15 Apr 2024 10:55:32 GMT

Thanks, Pavan. I'm all set now and I have a trained model.

Re: ERROR WHEN TRAINING DEEP MODEL

FirdevsGuzel — Mon, 06 May 2024 13:18:15 GMT

I'm getting the same error as you right now. Could you please explain how you fixed it?

Re: ERROR WHEN TRAINING DEEP MODEL

GBacon — Mon, 06 May 2024 13:27:31 GMT

@FirdevsGuzel I fixed the error by selecting the higher level folder "EvergreenTrain" that contained both the \images and \labels subfolders in my post below. I was incorrectly trying to add the subfolders rather than their parent folder.

Re: ERROR WHEN TRAINING DEEP MODEL

FirdevsGuzel — Mon, 06 May 2024 13:29:33 GMT

Thank you for your quick reply. I get the same life when I select the main folder. Do you have any suggestions on what I can do?

Re: ERROR WHEN TRAINING DEEP MODEL

FirdevsGuzel — Mon, 06 May 2024 13:32:50 GMT

My failed message

Traceback (most recent call last):
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 308, in execute
data_bunch = prepare_data(in_folders, working_dir=out_folder, **prepare_data_kwargs)
File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\learn\_data.py", line 1440, in prepare_data
raise Exception(
Exception: Could not infer dataset type. Please specify a supported dataset type or ensure that the path contains valid esri files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 390, in <module>
execute()
File "c:\program files\arcgis\pro\Resources\ArcToolbox\toolboxes\Image Analyst Tools.tbx\TrainDeepLearningModel.tool\tool.script.execute.py", line 384, in execute
del data_bunch
UnboundLocalError: local variable 'data_bunch' referenced before assignment

Re: ERROR WHEN TRAINING DEEP MODEL

GBacon — Mon, 06 May 2024 13:36:27 GMT

All I can think of if you haven't done it already, is to start a new Pro project and try again. I got stuck with these errors until I started over.

Re: ERROR WHEN TRAINING DEEP MODEL

FirdevsGuzel — Mon, 06 May 2024 13:41:04 GMT

thank you. I will try.

Have a nice day.