nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

GPU not working

AdrienMichez

Hi,

I'm trying to use GPU capabilities to train a deep learning model in arcgis pro using the 'train deep learning model' tool. I used the msi installation to install deep learning capabilities. It's working fine but is so slooooooow!

From other posts, I checked that Cuda is properly installed and that the python command torch.cuda.is_available() returns True.

I've activated GPU in environment/processor type but looking at task manager, my GPU does not seem to work hard, neither the CPU's.

I also check with the nvidia-smi monitoring tool which confirms that no running processes are found:

I'm using a GeForce GTX 770. Maybe this model is not compatible? It's not brand new nvidia

Best regards

Adrien

Trouver plus de publications taguées avec

Analysis

Deep learning

Image Mapping

Accepted answers

AdrienMichez

Thanks

@PavanYadav: yes I was setting environment just like that

I choose to buy a new GPU NVIDIA GeForce RTX 4090

That helped a lot 🙂

But I'm facing other problems... If you want to have a look here :

https://community.esri.com/t5/arcgis-image-analyst-questions/rare-habitat-mapping-with-deeplearning-nodata/m-p/1283713/highlight/true#M455

All comments

PavanYadav

@LeonelRivero_79

About #1, are you considering using the deep learning tools to detect objects as a GP service. While I'm not a web application expert, I'll consult with my colleagues for more insights.

About #2, we need to assess whether 15 hours is a reasonable processing time given the amount of data, GPU vRAM, and batch size you're using. The tips I shared in my previous post can help narrow down this. Are you seeing an optimal level of GPU vRAM utilization (greater than 80%)? If not, try increasing the batch size. If vRAM utilization is already close to 100%, consider increasing the GPU's vRAM. You can also use multiple GPUs on the same machine or scale up processing using ArcGIS Enterprise if this is something you would to often. (thinking about return on investment). If you're using ArcPy outside of ArcGIS, ensure that arcpy.env.processorType is set to 'GPU'. (by default the tool should use available supported GPU, but just in case you can try with and without if it is an issue)

About #3, please make sure the batch size is not too small. if you were using 4, maybe try 16. Please test different numbers and see for which number you get faster output. I understand when the batch size is too big, the tool might return an out of memory error.

I hope this helps.

Cheers!

LeonelRivero_79

@PavanYadav

I’m using ArcGIS Notebooks in ArcGIS Online with GPU support to train a deep learning model (MaskRCNN) and publish the package (.dlpk). The training and publishing work well, but object detection with DetectObjectsUsingDeepLearning in ArcPy takes about 15 hours, even with GPU.

Questions:

Web Applications: Is it possible to use the deep learning package in web applications for real-time object detection?
Optimization: How can I reduce the detection time? Is it normal for it to take this long even with GPU?
Recommended Workflow: What is the most efficient workflow for object detection with ArcGIS Online?

Thank you for your assistance.

Best regards,

PavanYadav

@AlejandraFigueroaMartínez

Possible Reasons for GPU Underutilization:

Small Batch Size: If the batch size is too small for the available GPU vRAM, the GPU may not be fully utilized. Try increasing the batch size to see if this improves performance.
CPU-GPU Synchronization: The GPU might be waiting for the CPU to send data, leading to fluctuating GPU usage. Monitor CPU and GPU usage to identify any synchronization issues.
Software Bug: as of today, we are not aware of any bugs related to GPU for this tool.
Small Input Rasters: If the input rasters are very small (e.g., 1024x1024), GPU/CPU will give you about the same performance.

Additional Tests for GPU Verification:

CPU vs. GPU Performance Comparison: A significant difference in processing time (3-4x to 20x) between CPU and GPU should be observed for large enough input rasters. Compare the processing times to determine if the GPU is working as expected.
Large Batch Size Test: Setting a very large batch size (e.g., 1024) should result in a "CUDA out of memory" error if the GPU is functioning correctly. If you see a "CPU out of memory" error, try a smaller batch size like 512.

AlejandraFigueroaMartínez

Hi @PavanYadav I have the same problem, I have an NVIDIA RTX 5000 graphics card, but when I perform object detection using deep learning, the card doesn't seem to be working properly. It does eventually complete the task, but it's incredibly slow.

PavanYadav

@AdrienMichez thanks for sharing. For the other issue, I discussed with my coworkers. I see one of them have already responded to you the thread. Hope, what she shared can help resolve issue.

AdrienMichez

Thanks

@PavanYadav: yes I was setting environment just like that

I choose to buy a new GPU NVIDIA GeForce RTX 4090

That helped a lot 🙂

But I'm facing other problems... If you want to have a look here :

https://community.esri.com/t5/arcgis-image-analyst-questions/rare-habitat-mapping-with-deeplearning-nodata/m-p/1283713/highlight/true#M455

PavanYadav

@AdrienMichez I am curious if you were setting GPU in the Processor Type

You may also want to try to check:
>>> import torch
>>> torch.cuda.is_available()

True

>>> torch.cuda.device_count()

>>> torch.cuda.current_device()

>>> torch.cuda.device(0)

<torch.cuda.device at 0x7efce0b03be0>

>>> torch.cuda.get_device_name(0)

'GeForce GTX 950M'

Even though with a 2GB GPU, you might not see much performance improvements. I just wanted to share the above, and you should be able to use it.

PavanYadav

@DanPatterson thank you for sharing the FAQs link.

I'm copying/pasting info from the link above so in case other users refer to this thread:

The recommended VRAM for running training and inferencing deep learning tools in ArcGIS Pro is 8GB. If you are only performing inferencing (detection or classification with a pretrained model), 4GB is the minimum required VRAM, but 8GB is recommended.

If you do not have the required 4–8GB VRAM, you can run the tools on the CPU, though the processing time will be longer.

@AdrienMichez looks like geforce-gtx-770 is 2GB memory size. (https://www.techpowerup.com/gpu-specs/geforce-gtx-770.c1856)

AdrienMichez

Yes I did

I think this is probably due to my old GPU but all the tests/checks found online came out positive so I don't have the confirmation that it is actually this issue

danretired

You checked the faq?

Deep learning frequently asked questions—ArcGIS Pro | Documentation

Rejoindre la conversation

Les membres connectés peuvent publier, suivre les mises à jour, et plus encore. Nouveau ici ? Inscrivez-vous gratuitement.

Liens rapides

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.