Hello! My name is Maja Kucharczyk and I was recently a Visiting Researcher at the University of Calgary, where I evaluated geospatial technologies (including small drones and AI-based damage mapping) as tools to support disaster and emergency management. Last year, my co-authors and I published a GeoAI workflow for automated roof damage delineation using ArcGIS Pro deep learning tools and post-storm aerial imagery from the Caribbean. In addition to publishing the study, we uploaded our data, tools, and trained models for anyone to access. We also wrote step-by-step instructions for reproducing the results and then created a StoryMap to visually summarize our work.
I proposed this project in 2018 as the final of three research studies in my PhD program at the University of Calgary. As I published my first two projects and began working on this one in 2020, little did I know the journey that was ahead of me. In addition to reading the research literature, I became familiar with deep learning by going through articles and tutorials that Esri was releasing, including a section on GeoAI in the first offering of its popular Spatial Data Science MOOC. After an initial round of trial and error, I submitted my preliminary workflow and results to the 2022 Esri Young Scholar Award and Esri Canada GIS Scholarship competitions. These awards led to numerous opportunities, including presenting at the 2022 User Conference, co-authoring a Q&A article with Krista Amolins in the Esri Canada Research & Education blog, and being a guest on the Geographical Thinking podcast with Guan Yue of Esri Canada. All this positive feedback and interest affirmed me as a GIS and remote sensing student and provided lasting motivation.
At my thesis defense in the spring of 2023, my PhD committee members made key recommendations for improving the workflow and increasing model accuracy. After passing my defense, I continued at the University of Calgary as a Visiting Researcher to further refine and publish this research. Fast forward to now as I am writing this: I recently returned from the Esri Developer & Technology Summit, where I witnessed our ArcGIS Living Atlas models highlighted and demoed at the plenary by Priyanka Tuteja of the GeoAI team!
In this article, I want to continue the tradition of sharing advice. Below are the top five GeoAI lessons I learned along the way. While not all these tips may work for you or be applicable, I hope they will encourage you to think critically and creatively! Here is a summary in case you don’t stick around for the entire article:
- Resample all input imagery to a fixed pixel size
- Comprehensively label your training images
- Test multiple model types
- Train single-class models for comparison with multi-class models
- Extend ArcGIS Pro deep learning tools using ModelBuilder or Python
Lesson #1: Resample all input imagery to a fixed pixel size
The aerial orthomosaics I used in my study had pixel sizes ranging from 0.02–0.15 m. When I used the Export Training Data For Deep Learning tool to export training tiles, the ground sample distance range resulted in excessive variety in terms of how damage objects and their surroundings were represented. In tiles that were exported from orthomosaics with a small pixel size (e.g., 0.02 m), the scene was too “zoomed in”, featuring only a portion of roof damage and limited surrounding context. At the other end of the spectrum with a larger pixel size (e.g., 0.15 m), the training tile was highly “zoomed out” and showed excessive surrounding context.
During my thesis defense, my committee encouraged me to try resampling all orthomosaics to a fixed pixel size to increase consistency in the training and inference data. I experimented by using the Resample tool to output select orthomosaics to pixel sizes ranging from 0.02–0.15 m. After exporting training tiles from these orthomosaics and examining their differences, I chose 0.05 m because it offered balance in terms of representing damage with sufficient detail and adequate surrounding context (Figure 1 and Figure 2). Once I resampled all training and testing orthomosaics to 0.05 m/px, I saw an improvement in model accuracy.
Figure 1. Example training tiles (sets of images and labels) exported using roof decking training polygons (yellow) and 0.05 m orthomosaics captured in (a–e) Dominica and (f–j) Sint Maarten. Copied from our publication (CC BY 4.0).
Figure 2. Example training tiles (sets of images and labels) exported using roof hole training polygons (red) and 0.05 m orthomosaics captured in (a–e) Dominica and (f–j) Sint Maarten. Copied from our publication (CC BY 4.0).
Lesson #2: Comprehensively label your training images
Another inconsistency in our data was that not all training images were comprehensively labeled. In other words, not all roof damage was being captured by our training polygons. I carefully reviewed the polygons and created additional ones to label roof damage that was initially missed. As of version 3.6, ArcGIS Pro features new training data review tools to help with the quality assurance and quality control (QA/QC) process.
In the context of labeling roof damage in post-storm aerial imagery, I learned that not all scenes are comprehensively delineable. For example, aerial images captured in urban areas in Dominica (e.g., Roseau) and Sint Maarten (e.g., Philipsburg) contained a high density of buildings with severely damaged roofs. Whereas some damage objects had discernable boundaries, other instances of damage were more complex with unclear boundaries (Figure 3 and Figure 4). Although these orthomosaics were not entirely delineable, I opted to label them as comprehensively as possible and retain them because it was important to include representation of urban areas in the training data.
Figure 3. Example training tiles from (a–e) Dominica and (f–j) Sint Maarten showing roof decking training polygons (yellow) and affected rooftops on which damage could not be comprehensively labeled. Copied from our publication (CC BY 4.0).
Figure 4. Example training tiles from (a–e) Dominica and (f–j) Sint Maarten showing roof hole training polygons (red) and affected rooftops on which damage could not be comprehensively labeled. Copied from our publication (CC BY 4.0).
We suspected that these complex scenes adversely impacted model training. As training progressed, the model learned to ignore non-delineable roof damage since it was not labeled in the tiles (Figure 3 and Figure 4). Because non-delineable roof damage is similar in appearance to our classes of delineable roof damage, the model may also have learned to ignore some forms of our classes.
When we expanded the training data using orthomosaics from the Bahamas and US Virgin Islands in which roof damage was more comprehensively labeled, we noticed an increase in model accuracy. These scenes featured buildings that were spread farther apart and had less complex roof damage that was clearly distinguishable from its surroundings (Figure 5 and Figure 6). The addition of these training tiles likely contributed to overall increases in the recall of roof decking from 0.73 to 0.83 and recall of roof holes from 0.52 to 0.81, as well as higher F1 scores in all four test areas (i.e., Dominica, Sint Maarten, the Bahamas, and US Virgin Islands).
Figure 5. Example training tiles exported using roof decking training polygons (yellow) and 0.05 m orthomosaics captured in (a–e) the Bahamas and (f–j) US Virgin Islands. Copied from our publication (CC BY 4.0).
Figure 6. Example training tiles exported using roof hole training polygons (red) and 0.05 m orthomosaics captured in (a–e) the Bahamas and (f–j) US Virgin Islands. Copied from our publication (CC BY 4.0).
Lesson #3: Test multiple model types
My preliminary workflow used Mask R-CNN, which is an object detection/segmentation model type. After my thesis defense, I started experimenting with other models. Since I began the project in 2020, Esri has incorporated numerous additional model types into its deep learning tools. These include transformer-based architectures (e.g., Mask2Former), which have been reported as being capable of learning more complex patterns in data than convolutional neural network-based architectures (e.g., Mask R-CNN).
I exported a subset of my training data using the Export Training Data For Deep Learning tool and used the subset to train each available pixel classification model type. Training can be performed using the Train Deep Learning Model tool or the ArcGIS API for Python in an ArcGIS Notebook or Jupyter Notebook (which is installed with ArcGIS Pro). The Mask2Former (MMSegmentation) model had the lowest validation loss, which prompted me to perform additional testing and eventually switch to using this model type in our workflow. Esri’s deep learning tools are constantly evolving, so experimenting with the latest models is highly encouraged. The model selection and hyperparameter tuning process can also be automated using the Train Using AutoDL tool.
Lesson #4: Train single-class models for comparison with multi-class models
During my thesis defense, my committee suggested that I try training single-class models (i.e., one for roof decking and one for roof holes) for comparison with my dual-class modeling approach. After exporting single-class training datasets and training each model, we observed higher accuracies from single-class models. The complexity of both roof damage classes was likely too high for one model to adequately learn. In other words, each roof damage class needed the sole focus of one Mask2Former model.
Table 1 summarizes accuracy metrics for the dual-class and single-class models we trained using the original training datasets derived from orthomosaics captured in Dominica and Sint Maarten. When switching to a single-class modeling approach, F1 scores increased from 0.73 to 0.79 for roof decking and from 0.62 to 0.66 for roof holes. Table 1 also shows the impact of expanding the training datasets with tiles from the Bahamas and US Virgin Islands. When single-class models were trained using the expanded training datasets, F1 scores increased from 0.79 to 0.88 for roof decking and from 0.66 to 0.80 for roof holes. These gains in accuracy were likely due to the expanded training datasets contributing comprehensively labeled tiles and representation of all four test areas (i.e., Dominica, Sint Maarten, the Bahamas, and US Virgin Islands). Figure 7 and Figure 8 show example roof damage predictions made by the best-performing models.
Table 1. Accuracy results for each trained model. The accuracy assessment summed the number of test image pixels categorized as true positive, false positive, and false negative. These sums were used to calculate common accuracy metrics, including precision, recall, and F1 score. Modified from our publication (CC BY 4.0).
Model | Precision | Recall | F1 Score |
Roof Decking Accuracy (Dual-Class Model, Original Training Dataset) | 0.95 | 0.59 | 0.73 |
Roof Hole Accuracy (Dual-Class Model, Original Training Dataset) | 0.91 | 0.47 | 0.62 |
Roof Decking Accuracy (Single-Class Model, Original Training Dataset) | 0.87 | 0.73 | 0.79 |
Roof Hole Accuracy (Single-Class Model, Original Training Dataset) | 0.90 | 0.52 | 0.66 |
Roof Decking Accuracy (Single-Class Model, Expanded Training Dataset) | 0.93 | 0.83 | 0.88 |
Roof Hole Accuracy (Single-Class Model, Expanded Training Dataset) | 0.78 | 0.81 | 0.80 |
Figure 7. Example true positive predicted polygons of roof decking (yellow) from test areas in (a–e) Dominica, (f–j) Sint Maarten, (k–o) the Bahamas, and (p–t) US Virgin Islands. Copied from our publication (CC BY 4.0).
Figure 8. Example true positive predicted polygons of roof holes (red) from test areas in (a–e) Dominica, (f–j) Sint Maarten, (k–o) the Bahamas, and (p–t) US Virgin Islands. Copied from our publication (CC BY 4.0).
Lesson #5: Extend ArcGIS Pro deep learning tools using ModelBuilder or Python
My preliminary workflow used the following ArcGIS Pro deep learning tools: Label Objects for Deep Learning, Export Training Data For Deep Learning, ArcGIS Notebook for model training, Detect Objects Using Deep Learning, and Compute Accuracy For Object Detection. To automate the generation, post-processing, and evaluation of model predictions, I was using ModelBuilder to chain together Detect Objects Using Deep Learning, Aggregate Polygons, and Compute Accuracy For Object Detection. ModelBuilder is an accessible, no-code way to visually build and document a workflow with data and ArcGIS Pro tools.
As our workflow evolved, we added an image preparation step that performed 3–5 geoprocessing tasks on each orthomosaic: band extraction (if needed), projection (if needed), resampling, clipping, and exporting. We also switched from an object-based model type (Mask R-CNN) to a pixel-based model type (Mask2Former), so our approach for generating, post-processing, and evaluating model predictions changed. Post-processing model predictions generated by the Classify Pixels Using Deep Learning tool involved multiple geoprocessing tasks. Similarly, we implemented a custom, multi-step accuracy assessment approach that differed from the one used by the Compute Accuracy For Pixel Classification tool.
After these modifications to the workflow, I realized there was a need to build custom ArcGIS Pro tools. Each tool would perform a major step in the workflow: Prepare Images, Export Training Data, Train Dual-Class/Single-Class Model, Delineate Roof Damage, and Calculate Accuracy. With this approach, I could consolidate the numerous geoprocessing tasks into their respective tools. Furthermore, sharing our workflow via custom tools would support reproducibility. For each script tool, most geoprocessing parameters were hard-coded into the script. The only user-facing parameters were the paths to input and output files and folders. Similarly, for each model-training notebook, the only modifications needed from the user involved the paths to the input training data folder and output trained model folder. Overall, our use of Python to create custom ArcGIS Pro tools: (1) automated our workflow, (2) allowed us to extend the functionality of the standard deep learning tools, and (3) supported the reproducibility of our workflow and results.
Final thoughts
The overarching lesson I took away from this multi-year journey is that developing a workflow is an iterative and collaborative process. Instead of aiming for perfection and a completely polished product, I learned to lean on others for their knowledge and advice. Presenting my progress at various stages throughout my PhD program allowed me to capture and implement feedback for incremental gains in model accuracy (as well as in my confidence). Eventually, I ended up with a published workflow and trained models I am very proud of. I wish you the same spirit of curiosity, tenacity, and collaboration in your GeoAI journey!
If you have any questions, feel free to comment below, connect with me on LinkedIn, or email me at maja.kucharczyk25@gmail.com.
