topic Re: Train deep learning model in multiple GPUs for Python API in ArcGIS Image Analyst Questions

Train deep learning model in multiple GPUs for Python API

JadedEarth — Sat, 19 Nov 2022 11:03:43 GMT

I'm using ArcGIS Pro v2.9.2 with Image Analyst license. I've trying to train a MaskRCNN model using multiple GPUs on a single machine but I can't seem to find a sample code. Most are distributed codes (multiple machines with multiple GPUs).

Would someone show me codes for training models for a single machine, multiple GPUs in a Python Window environment on ArcGIS Pro. I have my own training data generated using Image Analyst's Export training Data.

Appreciate any help.

Re: Train deep learning model in multiple GPUs for Python API

DanPatterson — Sun, 20 Nov 2022 02:41:58 GMT

I will move this thread to Imagery and Remote Sensing Questions

since your previous, closed thread went unresolved. Perhaps someone here from the Imagery team will have a solution

Re: Train deep learning model in multiple GPUs for Python API

JadedEarth — Wed, 19 Apr 2023 19:29:18 GMT

I can't find where this thread went in Imagery and Remote Sensing Questions. However, I have a follow-up with regards to using multiple GPUs in a single machine.

I'm using ArcGIS Pro version 3.1.4 with 7 GPUs. I have an ArcGIS Pro Online, named-instance license and an Image Analyst license.

From "Train Deep Learning Model" Image Analyst tool (aka tool.script.execute.py script), there is this script at the start:

#--------------------------------------------------------------------------------------------------------------------------------

if arcpy.env.processorType == "GPU" and torch.cuda.is_available() and arcpy.env.gpuId:
# use specific gpu if gpuId is specified, use all available gpus if no gpuID is specified

elif not arcpy.env.processorType:
# use all available gpus if processor type is not specified(default), gpuID is ignored in this case
arcgis.env._processorType = "GPU"

else:
arcgis.env._processorType = arcpy.env.processorType

#--------------------------------------------------------------------------------------------------------------------------------

It seems like the intent was that, if I specify the processor = "GPU" and I leave GPU ID blank, then I can use all the available GPUs in my machine. Here are the results of my trial settings for CPU/GPU on my machine:

Machine 1: 2 CPUs @ 28-core each; logical processors=56 cpus; 7 GPUs (1 RTX A6000 ID=4; and 6 RTX A4000 ID=0,1,2,3,5,6)

Settings 1:
Processor = Blank
GPU ID = Blank
Actual CPUs Used = 100% logical processors running (56 cores)
Actual GPU Used = 1 GPU Used (RTX A6000)

Settings 2:
Processor = GPU
GPU ID = Blank
Actual CPUs Used = 100% logical processors running (56 cores)
Actual GPU Used = 1 GPU Used (RTX A6000)

Settings 3:
Processor = GPU
GPU ID = 4
Actual CPUs Used = 100% logical processors running (56 cores)
Actual GPU Used = 1 GPU USED ID=1 (RTS A4000; Not the specified ID)

Settings 4:
Processor = CPU
GPU ID = Blank
Actual CPUs Used = 100% logical processors running (56 cores)
Actual GPU Used = None

Settings 5:
Processor = CPU
GPU ID = 4
Actual CPUs Used = 100% logical processors running (56 cores)
Actual GPU Used = None

Can someone at ESRI check why this is? Is this just a bug, or was this the intent for ArcGIS Pro Online licenses?

Appreciate any help.

Re: Train deep learning model in multiple GPUs for Python API

JadedEarth — Sat, 22 Apr 2023 10:29:13 GMT

Is anybody there? How can I get any technical help? I feel like I'm talking to myself here. Do I have to wait a year before someone from ESRI replies? And why is there only a reply button instead of "Post" button?

Re: Train deep learning model in multiple GPUs for Python API

DanPatterson — Sun, 23 Apr 2023 01:53:30 GMT

Contact Technical Support. Esri staff doesn't normally follow Community to answer technical support issues.

Re: Train deep learning model in multiple GPUs for Python API

JadedEarth — Mon, 24 Apr 2023 13:20:47 GMT

They just don't make it easy. I submitted a request, which means it goes to HQ in DC and hopefully I get a response after a week or two. Sigh...

Re: Train deep learning model in multiple GPUs for Python API

JadedEarth — Wed, 24 May 2023 13:42:11 GMT

Finally got through ESRI Tech support--after twists and turns. It turns out, this might be a bug in the system. They've placed this on their to-do list but couldn't tell me if this will be included in the next patch. This issue exists in version 2.9, 2.9.5, and 3.1.

Just so you know.

Re: Train deep learning model in multiple GPUs for Python API

PavanYadav — Fri, 22 Sep 2023 22:15:18 GMT

please see my response on https://community.esri.com/t5/arcgis-pro-questions/train-deep-learning-model-using-multiple-gpus-on/m-p/1221102

Cheers!

Pavan Yadav | Product Engineer - Imagery and AI
Esri | 380 New York | Redlands, 92373 | USA

https://www.linkedin.com/in/pavan-yadav-1846606/