Export Training Data for Deep Learning performance

633
6
Jump to solution
01-03-2022 12:41 PM
Labels (1)
MartyRyan
New Contributor III

I am running the Export Training Data tool in ArcGIS Pro 2.9.0, and it has been running for 13 days, reaching 81% on 12/29/21. This morning, it was at 28%. Geoprocessing is going on per the ArcGIS Pro Diagnostic Monitor. My output images folder has not had any images added to it since 7:00 pm EST on 1/1/22. 

Can anyone shed any light on what might be going on? I am exporting the training data to an external drive attached to my PC via USB. It has 3.5TB of available space, so it should accommodate my images. 
Any ideas would help....I have no idea how to proceed here. Thank you.

0 Kudos
1 Solution

Accepted Solutions
Kanin
by Esri Contributor
Esri Contributor

How is your machine's performance doing (CPU/RAM)? Also, could try saving the output on your local storage instead of external? (This can be a small sample area to help investigate further).

Best regards,
Kanin 

If this answer solved your question, please mark it as "Accept as Solution" to help others who have the same question.

View solution in original post

0 Kudos
6 Replies
FrankOuyang
Esri Contributor

Running for 13 days is a long time. Do you have lots of labels or the label area is large or the raster has multiple bands? In this case, I would suggest you to try to only export a few labels first and see what's the behavior of the software. If it does work and you are happy with the labels you have, then try to scale up. 

0 Kudos
MartyRyan
New Contributor III

1. I am using a 4-Band raster whose extent is a County

2. I have 1,067 labels 

The tool is still progressing although very slowly...it just hit 48% but still no new images are being populated in my output folder. The ArcGIS Pro Diagnostic Monitor shows that a Geoprocessing operation is in progress but the Tasks tab shows no information and the Counters tab shows no activity. I have no idea what is going on. 

0 Kudos
Kanin
by Esri Contributor
Esri Contributor

How is your machine's performance doing (CPU/RAM)? Also, could try saving the output on your local storage instead of external? (This can be a small sample area to help investigate further).

Best regards,
Kanin 

If this answer solved your question, please mark it as "Accept as Solution" to help others who have the same question.

0 Kudos
MartyRyan
New Contributor III

Performance seems fine - Pro is using  11% of CPU and 41% of memory. I am outputting these images to an external drive because in the past they have amounted to a couple hundred GB of space. 

I am very reluctant to stop this process and start it again with a smaller sample area. We have done that in the past - using smaller number of samples and the consistent feedback I have gotten is that we still do not have enough samples - the results in our smaller area were generally good, but this larger number of samples and the addition of a canopy height raster is the next step. Images have been generated (over 300,000 at this point). My issue is to find out what has seemingly halted that process. The progress bar in Pro is indication progress by very slowly. That, and the indicator lights in the Diagnostic Monitor that tell me a Geoprocessing task is underway are the only indications I have that "something" is going on. I just can't tell what it is. The "wait chain" document attached is what I am trying to figure out - can I end any of these processes that are "waiting" without crashing my whole project?

0 Kudos
MartyRyan
New Contributor III

Update: I discovered that the tool has output all the images it is going to as it matches a previous run of this tool using the same training samples. I have a case started with support as it appears that the tool is repeatedly querying the training samples over and over and appears to be stuck in this loop preventing it to finally finish.

0 Kudos
RichardDaniels
Regular Contributor

I was running the Export Training Data for Deep Learning and it reached 100%, then it started over again and it now up to 94%. Did ESRI give you any advice, e.g., do you have to kill the process, delete all chips and start over? Mine has been running for 12 hours. Could it be running once for each Zoom level in the raster image?

ESRI definitely needs to add some information messages to the results so we can actually see progress. 

0 Kudos