Detect Objects Using Deep Learning Error with new RTX 3060

[Hulseyj]

Error appears when I use "Detect Objects Using Deep Learning" in ArcGIS Pro with new RTX3060 GPU. The model used is usa_building_footprints.dlpk. I am using ArcGIS Pro 2.7.3. I have no issues running the model with my GTX 1650 Super GPU, same computer, same settings. It appears ArcPro engages the GPU as the application is shown running on the GPU: 

However, the program shows as "Running" but never progresses past 0%.

After about 30 minutes of "running", the following error message is returned:

I used the Input Parameters and Environment below. Again, it runs fine with the GTX 1650 Super GPU. All drivers are up to date on the new RTX3060. 

Thank you for help or troubleshooting ideas.

[Hulseyj]

Hi samaloysius4,

I created a new cloned environment following the install instructions. I am still getting the same error. Attached are the list and info files. Thank you

PREVIEW

[Paul_Christensen]

I am also seeing the same 002667 in Detect Objects...

I tried using the downloaded usa_building_footprints.dlpk and the online Living Atlas version.

Info and List reports appear identical to others.

RTX 3070

Side Note: Something to try for anyone experiencing verification errors when installing deep-learning-ampere-preview.

I could not get the deep-learning-ampere-preview to install because I kept getting tensorflow-base verification errors like "...package is corrupt, ... could not find file..." however I could manually explore to the file's location and see the file was there. 

It turns out that several of their files are buried a dozen folders deep and the combination of my long username and the packages being stored in %localappdata%, the file path was over the Windows 10 limit of 260 characters. I modified the windows filepath limits to allow paths that exceed 260 characters but that did not seem to affect conda.

I had to create a new location for packages closer to root in order to decrease the file path length and it finally worked.

[_Cartographer_]

Any update on fixes for the Ampere preview? I don't think anyone has been able to get it working yet unfortunately.

[borbalafarkas]

I also wonder whether there is any progress here.

The "preview" not just does not work but because if internal dependencies I was unable to fix it whatever I tried.

BTW, the preview is built with tensorflow 2.3.0 / tensorflow-gpu 2.3.0 (which is an upgrade from the 2.1.0 as was in the essentials pack). Isn't >=2.4.0 required (would be a better choice) for RTX 30 cards?

Anyway, I tried to upgrade to 2.4.0 and 2.5.0 (the ESRI conda page contains a 2.5.0 pack by ESRI, just a clue) but all failed. I mean, the updated 2.4.0 or 2.5.0 environments worked fine for other deep learning workflows (so the env is healthy otherwise) but do not work for the "standard" ESRI workflows (neither from Pro nor from Jupyter) as it was advertised to work. All my envs were cloned from arcgispro-py3 and contained the ESRI modules as much as dependencies permitted.

I used both RTX3050TI(4GB) and RTX3080(16GB) cards.

I also join to the above said:
                    - Is there anyone to confirm that the "preview" worked for them, please?

[borbalafarkas]

just FYI, (probably you already follow) this GitHub thread might be related:

Update for RTX 30 Series GPUs (CUDA 11) · Issue #17 · Esri/deep-learning-frameworks · GitHub

[borbalafarkas]

Well, winter is coming ...

I tried ArcGIS Pro 2.9 a few weeks ago and conda install -c esri deep-learning-essentials, as there was no install pack yet. Did not work on RTX3080. Nor did ampere preview.

Today I installed 2.9 on another notebook and found that there's a new 2.9 installer for deep learning packages. This time the code (ESRI MaskRCNN training) is running in Jupyter notebook (had no time to test further or directly from ArcGIS yet, but I mainly use Jupyter anyway). BTW this time it is tensorflow 2.5.1 in the env.

I also see that in the esri-build channel there is a 3.0 pack in prep.

Thought you wanted to know ...

[Hulseyj]

I am happy to report that Detect Objects Using Deep Learning is running successfully with my RTX 3060 after performing a clean install of ArcPro 2.9.0 and the 2.9 Deep Learning Libraries. I am seeing a significant improvement in speed as compared to my RTX 1650 Super. I have tested the Building Footprint Extraction deep learning model that is published by ESRI on the Living Atlas. I see better results in terms of speed when I use the default tile size environment settings (I remove the model argument: tile size 512). I have run the model in ArcPro and Jupyter notebook successfully. 

Thanks to all who have worked to solve the issue with using the deep learning tools on the new Ampere cards.

[Oliver_Morris]

I am getting this error - 

ERROR 999999: Something unexpected caused the tool to fail. Contact Esri Technical Support (http://esriurl.com/support) to Report a Bug, and refer to the error help for potential solutions or workarounds.

Tile size should be greater than zero.
Failed to execute (DetectObjectsUsingDeepLearning).

Running on a 24GB GPU - NVIDIA A10G

Any idea where the issues lies. Full script:

with arcpy.EnvManager(mask="raster1", processorType="GPU"):
out_classified_raster = arcpy.ia.DetectObjectsUsingDeepLearning(
in_raster="Orthomosaic.rgb.tif",
out_detected_objects=r"C:\Users\Administrator\Documents\ArcGIS\Projects\MyProject\MyProject.gdb\raster1_sam_out3",
in_model_definition=r"C:\Users\Administrator\Downloads\SAM.dlpk",
arguments="padding 512;batch_size 8;box_nms_thresh 0.7;points_per_batch 128;stability_score_thresh 0.6;min_mask_region_area 0",
run_nms="NO_NMS",
confidence_score_field="Confidence",
class_value_field="Class",
max_overlap_ratio=0,
processing_mode="PROCESS_AS_MOSAICKED_IMAGE"
)
out_classified_raster.save(None)