nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Train deep learning model in multiple GPUs for Python API

JadedEarth

I'm using ArcGIS Pro v2.9.2 with Image Analyst license. I've trying to train a MaskRCNN model using multiple GPUs on a single machine but I can't seem to find a sample code. Most are distributed codes (multiple machines with multiple GPUs).

Would someone show me codes for training models for a single machine, multiple GPUs in a Python Window environment on ArcGIS Pro. I have my own training data generated using Image Analyst's Export training Data.

Appreciate any help.

Najděte více příspěvků označených jako

Accepted answers

JadedEarth

Finally got through ESRI Tech support--after twists and turns. It turns out, this might be a bug in the system. They've placed this on their to-do list but couldn't tell me if this will be included in the next patch. This issue exists in version 2.9, 2.9.5, and 3.1.

Just so you know.

All comments

PavanYadav

please see my response on https://community.esri.com/t5/arcgis-pro-questions/train-deep-learning-model-using-multiple-gpus-on/m-p/1221102

Cheers!

Pavan Yadav | Product Engineer - Imagery and AI
Esri | 380 New York | Redlands, 92373 | USA

https://www.linkedin.com/in/pavan-yadav-1846606/

JadedEarth

Just so you know.

JadedEarth

They just don't make it easy. I submitted a request, which means it goes to HQ in DC and hopefully I get a response after a week or two. Sigh...

danretired

Contact Technical Support. Esri staff doesn't normally follow Community to answer technical support issues.

JadedEarth

Is anybody there? How can I get any technical help? I feel like I'm talking to myself here. Do I have to wait a year before someone from ESRI replies? And why is there only a reply button instead of "Post" button?

JadedEarth

I can't find where this thread went in Imagery and Remote Sensing Questions. However, I have a follow-up with regards to using multiple GPUs in a single machine.

I'm using ArcGIS Pro version 3.1.4 with 7 GPUs. I have an ArcGIS Pro Online, named-instance license and an Image Analyst license.

From "Train Deep Learning Model" Image Analyst tool (aka tool.script.execute.py script), there is this script at the start:

#--------------------------------------------------------------------------------------------------------------------------------

if arcpy.env.processorType == "GPU" and torch.cuda.is_available() and arcpy.env.gpuId:
# use specific gpu if gpuId is specified, use all available gpus if no gpuID is specified

elif not arcpy.env.processorType:
# use all available gpus if processor type is not specified(default), gpuID is ignored in this case
arcgis.env._processorType = "GPU"

else:
arcgis.env._processorType = arcpy.env.processorType

#--------------------------------------------------------------------------------------------------------------------------------

It seems like the intent was that, if I specify the processor = "GPU" and I leave GPU ID blank, then I can use all the available GPUs in my machine. Here are the results of my trial settings for CPU/GPU on my machine:

Machine 1: 2 CPUs @ 28-core each; logical processors=56 cpus; 7 GPUs (1 RTX A6000 ID=4; and 6 RTX A4000 ID=0,1,2,3,5,6)

Settings 1:
Processor = Blank
GPU ID = Blank
Actual CPUs Used = 100% logical processors running (56 cores)
Actual GPU Used = 1 GPU Used (RTX A6000)

Settings 2:
Processor = GPU
GPU ID = Blank
Actual CPUs Used = 100% logical processors running (56 cores)
Actual GPU Used = 1 GPU Used (RTX A6000)

Settings 3:
Processor = GPU
GPU ID = 4
Actual CPUs Used = 100% logical processors running (56 cores)
Actual GPU Used = 1 GPU USED ID=1 (RTS A4000; Not the specified ID)

Settings 4:
Processor = CPU
GPU ID = Blank
Actual CPUs Used = 100% logical processors running (56 cores)
Actual GPU Used = None

Settings 5:
Processor = CPU
GPU ID = 4
Actual CPUs Used = 100% logical processors running (56 cores)
Actual GPU Used = None

Can someone at ESRI check why this is? Is this just a bug, or was this the intent for ArcGIS Pro Online licenses?

Appreciate any help.

danretired

I will move this thread to Imagery and Remote Sensing Questions

since your previous, closed thread went unresolved. Perhaps someone here from the Imagery team will have a solution

Připojte se k diskusi

Přihlášení členové mohou přispívat, sledovat aktualizace a další. Jste tu noví? Zaregistrujte si bezplatný účet.

Rychlé odkazy

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.