How to train and use custom deep learning model in 3D Exploratory Analysis toolbar in Pro?

[kntr]

The 3D interactive object detection tool in the Exploratory Analysis toolbar in Arcgis Pro has pre-trained models for windows and doors detection from mesh. 

Ref: https://pro.arcgis.com/en/pro-app/latest/help/mapping/exploratory-analysis/interactive-object-detect...

Are there ways to train custom deep learning models to use with this tool? For example, road crack detection.

Or are there plans for Esri to add more pre-trained models into this tool?

Cheers!

[BobBooth1]

The Windows and Doors Extraction model item page (https://esriinc.maps.arcgis.com/home/item.html?id=8c0078cc7e314e31b20001d94daace5e) has a link to a developer documentation page on retraining the model:

https://developers.arcgis.com/python/latest/guide/retraining-windows-doors-extraction-model/

There is a crack detection model (for imagery) here:

https://esriinc.maps.arcgis.com/home/item.html?id=a9c3134e361e49a191efda169f5a337d

[dkutkevicius_hb]

Hello @BobBooth1 ,

This is very interesting. So if I understand your answer correctly, we can retrain Esri Windows and Doors Extraction model to detect other objects (like road cracks mentioned in original question)?

Also, are there any other ready to use .dlpk models for interactive detection tool in 3D scene?

[BobBooth1]

The Windows and Doors model is more suitable for detecting windows and doors, but the important point is that you can train, or in some cases, retrain, existing models. That sample code is about training the model to better detect specific types of windows and doors that are most common in your area, but a similar approach would apply for retraining a model to detect cracks.

The generic object detection model mentioned in this help topic may be retrainable to detect cracks:

https://pro.arcgis.com/en/pro-app/latest/help/mapping/exploratory-analysis/interactive-object-detect...

The tool accepts dlpk files where you select what model to use, so you should be able to train a model for your specific needs (crack detection in this case) and package it as a dlpk to use in the tool. 

Check out the crack detection model here:

https://www.arcgis.com/home/item.html?id=a9c3134e361e49a191efda169f5a337d

[URL edited]

I have not tested it with the 3D interactive object detection tool in the Exploratory Analysis toolbar, but it is probably worth experimenting with, to determine if it suits your needs.

Other pretrained object detection models are available:

https://livingatlas.arcgis.com/en/browse/?q=dlpk%20detection#d=2&q=dlpk+detection

[dkutkevicius_hb]

Thank you for detailed explanation. I will look into the possibility of retraining the model to suit our needs. I have tried to add other pretrained object detection models from living atlas but it seems that none of them can be added.

[BobBooth1]

I got some more information on this tool.

• You cannot download other models directly from within the tool. You have to download other models to your machine and then you can browse/choose the downloaded model using the browse button for the Model input.
• Only the following models are supported: FasterRCNN, YOLOv3, Single Shot Detector (SSD), and RetinaNet.

So, for a crack detection scenario, you would need to research which of those model types is likely to work well for cracks, download a pretrained model that is of one of those model types, do retraining following the general model of the developer doc, or try transfer learning, as shown in this tutorial (https://learn.arcgis.com/en/projects/improve-a-deep-learning-model-with-transfer-learning/).

Other tutorials that may be helpful in this process include:

https://learn.arcgis.com/en/projects/get-ready-for-deep-learning-in-arcgis-pro/

and

https://learn.arcgis.com/en/projects/train-a-model-using-automated-deep-learning/

[JerradDringman1]

Your link to the crack detection model doesn't appear to be working for me. Asks for a login that won't let me login. Is there another way to view what you've linked here?

Thank you.

[BobBooth1]

Oops, sorry.  It is publicly shared here:

https://www.arcgis.com/home/item.html?id=a9c3134e361e49a191efda169f5a337d

[BobBooth1]

I updated the URL in my post above too.

[fmuresu]

Hello everyone,

I'm jumping into this thread since it's one of the few that discusses the Object Detection tool within the Exploratory Analysis toolbar in ArcGIS.

I'm currently testing the tool in ArcGIS Pro 3.5.4 with the "Windows and Doors" model. Regardless of the model's actual performance, the resulting detected points always show an evident shift from the objects present in the image.

I managed to find another model that works with this tool, the "Pool Detection - USA" model, available at the following link:

https://ps-esriitalia.maps.arcgis.com/home/item.html?id=0e7dffe605c24bdfadf3c376bdf2d413

This model seems to be affected by the exact same problem. This leads me to believe that the issue is not with the model itself but rather with the too.

Please find two relevant screenshots attached.

Has anyone else experienced this same inconvenience?

Thanks!