Train Model Tool deletes images and labels

[MartyRyan]

I am running ArcGIS Pro 2.8.1 and had a failure in attempting to train a model. It ran for 42 minutes before failing and when I went to run it again, found that the output folder where I had stored the output from the Export Training Data for Deep Learning tool was gone. It had the images and labels generated, along with a couple other files. It was 226 GB of data lost, and the 3 days of exporting the training data was lost as well.

Has anyone run into this problem or know anything about it? Thanks. 

Marty

[DrVSSKiran]

I never face these type of issue. Lot of times tool got failed or ArcGIS Pro not responded when your train the model, exporting image chips or detect object using deep learning tool due to heavy load or compute your machine. In those scenarios eithet you have reduced chip size to run on same configuration machine or else you need to increase the gpu.

[MartyRyan]

Thank you

[DrVSSKiran]

I hope the issue is resolved. If it did, please mark the appropriate
response as solution (Accept as Solution). If the issue persists, please
provide more details on the issue.

[MartyRyan]

I won't know if the issue is resolved until I run the tool again with what I have discovered might be the solution - a different GPU ID in the Environment settings. I believe my PC has enough capabilities and am checking to ensure that is the case. Until the Export Training Data tool is complete and I train a model again, I don't have anything to add to this issue. I was simply looking to see if anyone else had experienced it or knew anything about it.

[DrVSSKiran]

Ok, test it and let me know if you face any issues.

[Anonymous User]

@MartyRyan you can test your workflow by exporting a small amount of data. Meanwhile can you provide us some more information about the type of model you were training ?