topic Re: Train Deep Learning Model Using Multiple GPUs on Windows machine in ArcGIS Pro Questions

Train Deep Learning Model Using Multiple GPUs on Windows machine

JadedEarth — Wed, 12 Oct 2022 14:14:03 GMT

I'm using ArcGIS Pro version 2.9. We have installed 6 GPUs in one machine but when I train deep learning model, there is no space to specify the IDs of the GPUs; only one space is available. However, it seems to randomly select which GPU it uses. Is this normal?

I'm not using Python. I'm just using Image Analyst to run the deep learning.

Any help is appreciated.

Re: Train Deep Learning Model Using Multiple GPUs on Windows machine

DanPatterson — Wed, 12 Oct 2022 15:58:17 GMT

GPU ID (Environment setting)—ArcGIS Pro | Documentation

does the tool you are using support the GPU environment setting?

Re: Train Deep Learning Model Using Multiple GPUs on Windows machine

JadedEarth — Wed, 12 Oct 2022 19:42:34 GMT

Yes. The deep learning library has Environment parameters that allow you to specify processor type. But it only has one slot for GPU processor. I'm wondering if you can place comma for each ID, like in python.

Re: Train Deep Learning Model Using Multiple GPUs on Windows machine

JadedEarth — Thu, 13 Oct 2022 15:13:11 GMT

Since I can't get this to work in Image Analyst, I'm switching to arcgis.learn using Jupyter Notebook interface in ArcGIS Pro.

Does anybody have sample Python codes to train model using multiple GPUs and using Data Parallelism?

Re: Train Deep Learning Model Using Multiple GPUs on Windows machine

JadedEarth — Fri, 14 Oct 2022 13:00:20 GMT

Further research: I really need to use Python codes for now; not only that, I need to use distributed data parallelism to be able to use multiple GPUs. This seems to be quite complicated. I'm still researching and will test whatever I find in literature. This is not a solution but I'll end this thread here.

Re: Train Deep Learning Model Using Multiple GPUs on Windows machine

JeffreyHolycross — Mon, 06 Mar 2023 14:35:28 GMT

@JadedEarth did you see the sample code here? I'm not sure when it was posted relative to your question, but I believe it was available last year: Train arcgis.learn models on multiple GPUs | ArcGIS API for Python

Re: Train Deep Learning Model Using Multiple GPUs on Windows machine

PavanYadav — Fri, 22 Sep 2023 20:31:09 GMT

@JadedEarth

At upcoming v3.2, by default, the Train Deep Learning tool is planned to use all available GPUs on a single machine when the Model Type is set to one of the following:

- ConnectNet
- Feature classifier
- MaskRCNN
- Multi Task Road Extractor
- Single Shot Detector
- U-Net
To use a specific GPU, you can use the GPU ID environment.
between 2.8 and 3.1, the tool could use multiple GPUs by default but the feature was not certified and thus not documented.

You could use API: https://developers.arcgis.com/python/guide/utilize-multiple-gpus-to-train-model/ if ArcGIS Pro's current support does not meet your need.

We hope to support multiple GPUs on single machine for inferencing as well in upcoming releases.

A side note: We have an ArcGIS Image Analyst dedicated community and here you might get response sooner.

Cheers!

Pavan Yadav | Product Engineer - Imagery and AI
Esri | 380 New York | Redlands, 92373 | USA

https://www.linkedin.com/in/pavan-yadav-1846606/

Re: Train Deep Learning Model Using Multiple GPUs on Windows machine

JadedEarth — Tue, 31 Oct 2023 14:00:58 GMT

I need the HED edge detection model. This is the only one that works with field boundary extraction. Is this not going to be included in this update? If not, then I'm still out of luck. I've tried all the other models you mentioned for my work but only HED model comes up with reasonable (not perfect) results.

I do hope you include HED among the multiple GPU supports since both USDA and EPA really need this.

I appreciate your assistance.