Train Model Tool deletes images and labels

Question

I am running ArcGIS Pro 2.8.1 and had a failure in attempting to train a model. It ran for 42 minutes before failing and when I went to run it again, found that the output folder where I had stored the output from the Export Training Data for Deep Learning tool was gone. It had the images and labels generated, along with a couple other files. It was 226 GB of data lost, and the 3 days of exporting the training data was lost as well.

Has anyone run into this problem or know anything about it? Thanks.

Marty

Anonymous User · Answer

@MartyRyan you can test your workflow by exporting a small amount of data. Meanwhile can you provide us some more information about the type of model you were training ?

DrVSSKiran · Answer

Ok, test it and let me know if you face any issues.

MartyRyan · Answer

I won't know if the issue is resolved until I run the tool again with what I have discovered might be the solution - a different GPU ID in the Environment settings. I believe my PC has enough capabilities and am checking to ensure that is the case. Until the Export Training Data tool is complete and I train a model again, I don't have anything to add to this issue. I was simply looking to see if anyone else had experienced it or knew anything about it.