Export Training Data for Deep Learning performance

[MartyRyan]

I am running the Export Training Data tool in ArcGIS Pro 2.9.0, and it has been running for 13 days, reaching 81% on 12/29/21. This morning, it was at 28%. Geoprocessing is going on per the ArcGIS Pro Diagnostic Monitor. My output images folder has not had any images added to it since 7:00 pm EST on 1/1/22. 

Can anyone shed any light on what might be going on? I am exporting the training data to an external drive attached to my PC via USB. It has 3.5TB of available space, so it should accommodate my images. 
Any ideas would help....I have no idea how to proceed here. Thank you.

[FrankOuyang]

Running for 13 days is a long time. Do you have lots of labels or the label area is large or the raster has multiple bands? In this case, I would suggest you to try to only export a few labels first and see what's the behavior of the software. If it does work and you are happy with the labels you have, then try to scale up. 

[MartyRyan]

1. I am using a 4-Band raster whose extent is a County

2. I have 1,067 labels 

The tool is still progressing although very slowly...it just hit 48% but still no new images are being populated in my output folder. The ArcGIS Pro Diagnostic Monitor shows that a Geoprocessing operation is in progress but the Tasks tab shows no information and the Counters tab shows no activity. I have no idea what is going on. 

[Kanin · Accepted solution]

How is your machine's performance doing (CPU/RAM)? Also, could try saving the output on your local storage instead of external? (This can be a small sample area to help investigate further).

Best regards,
Kanin 

If this answer solved your question, please mark it as "Accept as Solution" to help others who have the same question.

[MartyRyan]

Performance seems fine - Pro is using  11% of CPU and 41% of memory. I am outputting these images to an external drive because in the past they have amounted to a couple hundred GB of space. 

I am very reluctant to stop this process and start it again with a smaller sample area. We have done that in the past - using smaller number of samples and the consistent feedback I have gotten is that we still do not have enough samples - the results in our smaller area were generally good, but this larger number of samples and the addition of a canopy height raster is the next step. Images have been generated (over 300,000 at this point). My issue is to find out what has seemingly halted that process. The progress bar in Pro is indication progress by very slowly. That, and the indicator lights in the Diagnostic Monitor that tell me a Geoprocessing task is underway are the only indications I have that "something" is going on. I just can't tell what it is. The "wait chain" document attached is what I am trying to figure out - can I end any of these processes that are "waiting" without crashing my whole project?

[MartyRyan]

Update: I discovered that the tool has output all the images it is going to as it matches a previous run of this tool using the same training samples. I have a case started with support as it appears that the tool is repeatedly querying the training samples over and over and appears to be stuck in this loop preventing it to finally finish.

[RichardDaniels]

I was running the Export Training Data for Deep Learning and it reached 100%, then it started over again and it now up to 94%. Did ESRI give you any advice, e.g., do you have to kill the process, delete all chips and start over? Mine has been running for 12 hours. Could it be running once for each Zoom level in the raster image?

ESRI definitely needs to add some information messages to the results so we can actually see progress. 

[OddbjornLarsen1]

Hi, im having the same problem, the tool hangs at 100% and it does not create any more images or the metadata files. Im doing the export from a mosaic of multiple RGB images. Also tried to test with just two rasters combinging them using "Mosiac to new raster". Tried multiple options when creating mosaic dataset, but no luck so far. Left it working through the night (over 12 hours) but was still stuck at 100% in the morning. Images are saved to local SSD, rasters are also stored on that SSD. Im using classified tiles. When i left it over night, the memory usage increased to over 100 GB, so might be a memory leak somewhere? Also tried the "Export training data for deep learning" tool directly from the toolbox and also via Imagery -> Label objects for Deep learning. This tool has worked before with classified tiles and mosaic datasets. 

[PavanYadav]

hello @MartyRyan @OddbjornLarsen1 , 

I am aware of a known issue when the Export Training Data tool take unexpectedly very long time. We will try to address it in an upcoming release but there are workarounds, please continue reading for more details about the issue and workaround 

Issue: Export Training Data for Deep Learning takes unexpectedly very long time when the label features are sparse; it also reach to 100% in a few minutes but continue to work for several minutes (or, hours or days) depending on the inputs. In below case, there are 1600 label features. The tool will export ~90 items and then stop for a long time, then goes to another extent and exports......it may take hours or days. 

About my test case:
There are 1600 labels features and they are spread over entire USA but geographically  they can be clustered in 12 areas.

Workaround 1 You can use current extent for each areas and export chips one by one for each areas. You can either append existing folder or use multiple folder

Workaround 2: for each sample areas create a mask feature, with 12 rectangles around the 12 areas in the map above. Use this mask feature class as input to the tool.

Note: We have an Arcgis Image Analyst dedicated community and here you might get response sooner.

Cheers!

Pavan Yadav | Product Engineer - Imagery and AI
Esri | 380 New York | Redlands, 92373 | USA

https://www.linkedin.com/in/pavan-yadav-1846606/ 

[OddbjornLarsen1 · Accepted solution]

Thanks, solution #2 seems to work for me. My mosaic was similar to yours with several features spread around Norway. I just used the footprint from my mosaic as input for "Mask Polygon Features" and the tools seems to work now. Still that bug with the progress jumping to 100% after a few minutes, but images and masks are being created 🙂