AppStudio and TensorFlow Lite: Machine Learning on Edge

[ErwinSoekianto]

AppFramework DeepLearning Plugin is released as a beta feature in AppStudio 5.2, see the API Reference. 

==========================

TensorFlow is an open-source deep learning framework that provides powerful state-of-the-art offline capabilities. It provides low-level interfaces for defining tensors and derives inferences. It is highly reliable and provides scalability of computation across machines and large datasets. 

ArcGIS AppStudio uses TensorFlow Lite for its machine learning capability. TensorFlow Lite is a C++ library that is optimized to be used for mobile and IoT devices. In ArcGIS AppStudio, it is supported on all platforms (iOS, Android, Windows, MacOS, and Linux). It is compatible with tflite model file and only supports supervised learning. You can perform Image Classification which allows you to identify the main characteristics of an image to determine a class, and Object Detection which displays bounding boxes around multiple features in an image or video output. Both of these will work offline as long as we have the model file stored in the device. 

 

Starting at ArcGIS AppStudio 4.4, we introduced the TensorFlow plugin in AppFramework that provides a mechanism to work with a machine learning model. Currently, it is available only as a beta feature and it is subjected to change. AppFramework TensorFlow plugin introduces two components, ImageAnalyzer for static image analysis and ImageAnalysisFilter for real-time video analysis filtering. 

import ArcGIS.AppFramework.DeepLearning 1.0
...

ImageAnalyzer {
  id: imageAnalyzer

  debug: true
  modelSource: "model/model.tflite"
  classNames: "model/labels.txt"
  minimumScore: minimumScoreValue

  onFinished: {
    console.log("Results:", JSON.stringify(results, undefined, 2));
    resultsView.model = results;
    resultsListView.model = results;
  }

  onError: {
    console.error("Error:", message);
  }
}

In the code sample above, you can see that we have an import statement for TensorFlow from AppFramework.Labs, and then we have the ImageAnalyzer component. ImageAnalyzer component requires only three inputs, 

• modelSource. It is pointing to the tflite model file.
• classNames. It is pointing to a text file that contains the classes where the model will verify against the classes and then will reflect in the output.
• minumumScore. This is an optional property to set the minimum score of the result. 

Here is the demo of how the app looks like, 

Please refer to the sample "Deep Learning" in AppStudio Desktop and Player to see how it works and please refer to this documentation on how to train a TensorFlow model using ArcGIS API for Python using ArcGIS Notebook. 

Learn more about AppStudio:

• New to AppStudio? Check out this website to learn more about what ArcGIS AppStudio can do to you and your organization.
• Check out our recent workshop recordings to get started with creating native apps using AppStudio.
• Let us know about your creations built using AppStudio to be featured in AppStudio Showcase Gallery.
• Provide your ideas and suggestions to us on ArcGIS Ideas site.
• Follow us on Twitter @ArcGISAppStudio to keep up-to-date on the latest information.
• The AppStudio team periodically hosts workshops and webinars; please sign up for the AppStudio newsletter if you are interested in information regarding AppStudio events.