topic Re: Train Deep Learning Model stuck on Epoch 1 in ArcGIS Pro Questions

Train Deep Learning Model stuck on Epoch 1

MartynasBielinis — Wed, 05 Aug 2020 07:36:02 GMT

I am trying to run the Train Deep Learning Model tool on Pro 2.6 using single-shot detection and resnet34, but it seems to be stuck on Epoch 1. The progress bar is at 0% for hours with no change. The messages show text for "Training Loss" and "Validation Loss" but nothing after that.

Anyone deal with a similar problem?

Re: Train Deep Learning Model stuck on Epoch 1

jdetka — Sat, 26 Dec 2020 18:17:26 GMT

I am having the same problem but with Pro 2.7. Did you ever find a solution?

Re: Train Deep Learning Model stuck on Epoch 1

MartyRyan — Thu, 04 Mar 2021 12:59:02 GMT

I am having the same issue - Epoch 1 shows no progress after 18 hours. I am running this on my CPU, which I know makes for slow processing. (This is a test to see what my CPU will do.) The only indication I have of progress is that Task Manager shows my ArcGIS Pro project using almost 100% of my CPU and the Power usage trend shows "Very high". I am using a .5 m resolution image, so I realize this may take a long time. It took 33 hours to export my training data on this same image and computer.

Re: Train Deep Learning Model stuck on Epoch 1

MartyRyan — Fri, 05 Mar 2021 12:48:50 GMT

I discovered that (at least in my case) there doesn't seem to be an error - I am using only my CPU and so progress is slow...I am now at Epoch 3 (10%) - I started training at 2:00 PM Wednesday, 3/3.

Re: Train Deep Learning Model stuck on Epoch 1

MartyRyan — Fri, 12 Mar 2021 15:52:20 GMT

I was having the same issues with slowness - in my case, it was not an error, just where I was processing. We set up a remote PC with an 8 GB NVIDIA GPU (Quadro M4000). It made more progress in 1 hour than in 28 hours on my CPU. Comparable times for Epoch to Epoch progress: Start to Epoch 2 at 5% - 19 hrs. CPU / 20 minutes GPU...Epoch 2 at 5% to Epoch 3 at 10% - 9.5 hours CPU / 4minutes GPU. I got a trained model of 20 epochs in 3 hours.