How to train and use custom deep learning model in 3D Exploratory Analysis toolbar in Pro?

[kntr]

The 3D interactive object detection tool in the Exploratory Analysis toolbar in Arcgis Pro has pre-trained models for windows and doors detection from mesh. 

Ref: https://pro.arcgis.com/en/pro-app/latest/help/mapping/exploratory-analysis/interactive-object-detect...

Are there ways to train custom deep learning models to use with this tool? For example, road crack detection.

Or are there plans for Esri to add more pre-trained models into this tool?

Cheers!

[BobBooth1]

The Windows and Doors Extraction model item page (https://esriinc.maps.arcgis.com/home/item.html?id=8c0078cc7e314e31b20001d94daace5e) has a link to a developer documentation page on retraining the model:

https://developers.arcgis.com/python/latest/guide/retraining-windows-doors-extraction-model/

There is a crack detection model (for imagery) here:

https://esriinc.maps.arcgis.com/home/item.html?id=a9c3134e361e49a191efda169f5a337d

[dkutkevicius_hb]

Hello @BobBooth1 ,

This is very interesting. So if I understand your answer correctly, we can retrain Esri Windows and Doors Extraction model to detect other objects (like road cracks mentioned in original question)?

Also, are there any other ready to use .dlpk models for interactive detection tool in 3D scene?

[BobBooth1]

The Windows and Doors model is more suitable for detecting windows and doors, but the important point is that you can train, or in some cases, retrain, existing models. That sample code is about training the model to better detect specific types of windows and doors that are most common in your area, but a similar approach would apply for retraining a model to detect cracks.

The generic object detection model mentioned in this help topic may be retrainable to detect cracks:

https://pro.arcgis.com/en/pro-app/latest/help/mapping/exploratory-analysis/interactive-object-detect...

The tool accepts dlpk files where you select what model to use, so you should be able to train a model for your specific needs (crack detection in this case) and package it as a dlpk to use in the tool. 

Check out the crack detection model here:

https://esriinc.maps.arcgis.com/home/item.html?id=a9c3134e361e49a191efda169f5a337d

I have not tested it with the 3D interactive object detection tool in the Exploratory Analysis toolbar, but it is probably worth experimenting with, to determine if it suits your needs.

Other pretrained object detection models are available:

https://livingatlas.arcgis.com/en/browse/?q=dlpk%20detection#d=2&q=dlpk+detection

[dkutkevicius_hb]

Thank you for detailed explanation. I will look into the possibility of retraining the model to suit our needs. I have tried to add other pretrained object detection models from living atlas but it seems that none of them can be added.

[BobBooth1]

I got some more information on this tool.

• You cannot download other models directly from within the tool. You have to download other models to your machine and then you can browse/choose the downloaded model using the browse button for the Model input.
• Only the following models are supported: FasterRCNN, YOLOv3, Single Shot Detector (SSD), and RetinaNet.

So, for a crack detection scenario, you would need to research which of those model types is likely to work well for cracks, download a pretrained model that is of one of those model types, do retraining following the general model of the developer doc, or try transfer learning, as shown in this tutorial (https://learn.arcgis.com/en/projects/improve-a-deep-learning-model-with-transfer-learning/).

Other tutorials that may be helpful in this process include:

https://learn.arcgis.com/en/projects/get-ready-for-deep-learning-in-arcgis-pro/

and

https://learn.arcgis.com/en/projects/train-a-model-using-automated-deep-learning/