nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Detect Objects Using Deep Learning on 0% after over 15 hours

HKrell

Good morning,

I am trying to run the Detect Objects Using Deep Learning tool to detect solar panels in NAIP imagery. I am using ArcGIS Pro 2.7, and I have selected one small zip code area to detect the objects. However, the tool has been running for over 15 hours and is still at 0% completed. My CPU is at almost 100%, so the tool is definitely running. Has anyone else had this problem or knows any solutions? Please help if you can!

Find more posts tagged with

Analysis

Desktop

Accepted answers

All comments

AlexaVerónicaGarcíaBarrantes

Im having the same issue, Im trying to detect objects and I ran the tool for more than 6 hours and it didnt do anything, Im noticing that the tool works in very small areas but when I try to run it in larger ones, like 1 hectare, it totally freezes. I have a Asus, Nvidia RTK, its supposed to be about enough. Id appreciate any input.

P.D. starting to lose my mind over this

MartyRyan

Update: I have more questions regarding GPU\CPU usage after a failure of the Train Model tool. I am again Exporting Training Data for Deep Learning. My Train Model failure was shown as:

RuntimeError: cuda runtime error (101) : invalid device ordinal at ..\torch\csrc\cuda\Module.cpp:59

I chose GPU for Processor Type and "1" for GPU ID: 1 (for the NVIDIA GPU as shown below)

The tool ran for 42 minutes before showing this error. I have tried to ensure I am identifying the correct GPU in the Train Model tool - I thought I chose "1" correctly. However, when I check using nvidia-smi in Command Prompt, I get:

It appears here that the GPU is identified as "0".

How do I reconcile this apparent conflict? How do I ensure the NVIDIA GPU is being used most appropriately and efficiently? I am getting very slow performance in the Export Training Data tool (about 1% progress every hour) and am not sure my NVIDIA GPU is being used nearly enough - it's usage never goes above 11% or so. I need to know the correct GPU setting when I train my next model.

Thanks for any help

MartyRyan

Tim,

Sorry I haven't responded sooner - I checked and I am seeing this when running that code

Using Device: cuda
Device Name: Quadro M4000

RyanKammerer

Hi Tim,

Can you point me to where I can get the Pro 2.8.1 patch? I'm just running the 2.8 version from Github.

Thanks!

DylanT_EsriCanada

Another couple of things you could try:

# bands in raster
reduce batch size to 1
confirm which number your nvidia card is (might not be what you expect) - there is a command line prompt for this but I don't remember what it is, maybe someone else here does

Tim_McGinnes

Hey @MartyRyan

First of all, make sure you have installed the Pro 2.8.1 patch that just came out - it fixes an issue with Deep Learning and GPUs.

Pro will automatically drop back to using the CPU if the GPU is not detected. First find out your GPU model and memory - easiest way is in the Nvidia Control Panel. Click on the System Info in the lower left. Here's mine:

The important info is the card model and dedicated memory. The requirements section on the Esri Deep Learning Github site recommends minimum 2Gb dedicated memory (but you really need at least 4Gb to do anything serious). You also need CUDA Compute Capability 3.5 minimum, 6.1 or higher recommended. Find your GPU on the Nvidia CUDA GPU page and check your capabilities. Here's mine:

If your card meets the requirements, then I would recommend updating to the latest graphics card drivers if they haven't been updated in a while. Now we'll make sure it is recognised in ArcGIS Pro. Open Pro and open a new Notebook. Paste the following code into a cell and run it.

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using Device: ' + str(device))
if device.type == 'cuda':
    print('Device Name: ' + torch.cuda.get_device_name())

If the output says cuda and prints your graphics card model you should be ok. If it says cpu then it wasn't recognised. Here's mine:

See how you go. There could be technical problems with the card or drivers, or even issues with your Pro install.

Tim_McGinnes

In the tool output can you click on the Messages tab and send back what it is saying (make sure you copy all of them). To confirm that the model is ok I would suggest you zoom right in on about 10 houses, set the tool to that extent, set the batch size to 1 and run it with CPU. If that works then just change the processor type to GPU and run it again. Let me know how it goes.

Another option would be to download a pre-trained Esri model and make sure it works ok. Try Building Footprint Extraction - USA and see how you go using the same procedure as above. I am going to respond to Marty above with some GPU troubleshooting tips - might be worth trying those as well.

HKrell

Thank you for your advice @Tim_McGinnes . I changed my extent, switched over to GPU, and lowered my batch size to 4. However, now I am returning error 999999, something unexpected caused the tool to fail. I am confused on why it would fail now if it was running before, even if it was running slowly. I would appreciate any suggestions! error 999999.PNG

MartyRyan

Tim, I have a similar related question regarding CPU/GPU performance. I'm running the Train Model for Deep Learning tool in ArcGIS Pro 2.8 and while setting the GPU as Processor Type, my Task Manager shows almost no activity for GPU and huge activity in CPU and Ethernet. (I'm using Remote Desktop to connect to a PC with an NVIDIA graphics card (which we use specifically for the NVIDIA capability). I am at Epoch 1 (0%) for 3 hours and the activity in Task Manager tells me something is going on, it sure seems that I am getting very little use of the GPU with its NVIDIA card.

Thanks for any input you or anyone else may have.

Tim_McGinnes

Even though you have selected a zip code, the tool doesn't know that - it is trying to process the entire image. You have a couple of options: Clip out just the area you want to process and use that in the tool, or use the Extent settings in the Environments tab. The easiest option may be to zoom in to your required area and choose Current Display Extent.

Using the CPU for deep learning will be really slow anyway. Do you have a dedicated graphics card (preferably NVidia)? If so, in the Environments tab you should set the Processor Type to GPU. You may also have to reduce the Batch Size parameter according to the amount of memory available on your graphics card as well (probably start at 4 and reduce if you get any out of memory errors).

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.