System has two GPUs in it that are verifiable via DxDiag and the Nvidia Control Panel
GPU 0: WDDM
GPU 1: TCC
System environment variables:
CUDA_VISIBLE_DEVICES=1
Was running Viewshed 2 just fine yesterday, suddenly stopped recognizing GPU 1 and kept trying to use GPU 0 which was being used to show display; obviously this meant doing any display wise work was not optional.
In the Nvidia Control Panel, set GPU 0 Usage Mode to "Dedicate to graphics tasks" and this forces Viewshed 2 to use CPU.
Is there a setting somewhere that ArcPro would store erroneous CUDA_VISIBLE_DEVICES flags?
We cannot figure out why the application is saying there is no GPU available in TCC and specified when it clearly is.
Could you try removing the CUDA_VISIBLE_DEVICES variable ? Viewshed 2 should automatically find TCC card for computation.
Please also make sure the graphics driver for the TCC GPU is up to date.
I believe that is what we ended up doing. We followed these instructions to begin with:
GPU processing with Spatial Analyst—Help | ArcGIS Desktop
Then we read your blog and the NVIDIA programming guide:
Are you getting GPU errors while executing the Viewshed 2 tool?
Programming Guide :: CUDA Toolkit Documentation
Then we updated the Windows TDR settings by reading:
Timeout Detection and Recovery (TDR) | Microsoft Docs
TDR Registry Keys | Microsoft Docs
Both of which said to add CUDA_VISIBLE_DEVICES. However, we later noticed that the first one notes the following:
"In the case of multiple GPUs in your system, the first GPU in the TCC (Tesla Compute Cluster) driver mode will be used by default. If there is no GPU available in the TCC driver mode, the first GPU (with index 0) will be used, unless specified otherwise."
So we reset the system (put all settings and drivers back to original states), then we changed the mode of the second GPU to TCC, then opened our NVIDIA control panel and went to:
Manage GPU Utilization -> Check "Use for compute needs" on TCC GPU
Manage 3D Settings -> Global Settings -> CUDA - GPUs -> Ensure TCC card was only card selected
What we noticed with ArcGIS Pro 2.0.0 was that even if the NVIDIA control panel Global Settings and ArcGIS Pro Settings (in the program specific pane) showed the TCC GPU as the available CUDA GPU, ArcGIS Pro would issue a noticed that it failed to located a GPU and would send it to the primary GPU in WDDM that we were using to display graphics.
To resolve we reset the system, as noted and performed these steps inline: