ArcGIS Velocity: Extending real-time data ingestion support using the new gRPC feed type
Most feeds in ArcGIS Velocity connect to real-time data by either subscribing to a remote data source such as Kafka, MQTT, or Azure Event Hub, or polling a data source such as a third-party provider API.
ArcGIS can also receive real-time data on a REST endpoint by leveraging the HTTP Receiver feed type. And in the December update of Velocity, there is a new feed type that can be used to receive data: gRPC.
The gRPC feed type creates a built-in gRPC server endpoint that can be used to receive data sent from remote gRPC client applications. Client applications can be developed in a variety of languages, as supported by the gRPC framework. The gRPC feed type is currently supported in Beta and enables developers to become familiar with implementing gRPC clients for Velocity.
gRPC uses a .proto file to define the communication between gRPC clients and services. The .proto file can be compiled in numerous languages to produce native client libraries. Clients can connect to services created from the same .proto file regardless of the languages used.
Why gRPC?
gRPC is a modern, lightweight, high-performance framework that depends on a contract-first approach to API development. It is language agnostic and allows for implementation in a wide variety of development environments.
When to use gRPC?
- When you need to ingest real-time data into ArcGIS from providers or in data formats that ArcGIS Velocity does not support out-of-the-box: as with the HTTP Receiver feed type, you can develop a client application to push data into an ArcGIS feed in a supported format
- When you need to implement custom clients to stream data into ArcGIS: gRPC supports both streaming and unary calls, while pushing data in over an HTTP endpoint only supports unary calls.
- When you have environments with clients developed in diverse languages and all need to connect to ArcGIS Velocity.
- When efficient communication is paramount such as in CPU-constrained or limited bandwidth environments.
gRPC in Velocity
The most common approach for implementing a gRPC-based ingestion solution will be to first configure a gRPC feed in Velocity. The resulting feed details page will provide essential connection information to be leveraged when creating and deploying client applications.
Configure a gRPC feed much like you would other feed types. It is similar to the HTTP Receiver feed type in that you must supply sample data from which to derive a schema or manually build the schema field-by-field.
The feed details page will provide three key values that you will need when developing gRPC clients: the gRPC endpoint URL, gRPC endpoint header path and whether the feed is configured with ArcGIS OAuth 2.0 authentication.
With these connection parameters, you will select the development environment of your choice and write the client code to connect to your gRPC feed and send it data. That development effort will require the velocity_grpc.proto file. The .proto file is a key element of a gRPC implementation in that it defines the messages that gRPC clients and services may exchange with one another and serves as the contract under which they communicate. This file must be compiled for your language of choice to generate the client code classes. You can compile it with Google’s Protocol Compiler, but some integrated development environments will compile it when you build your code. When your client application is ready you will provision it with the . Don’t forget to configure your client to use an ArcGIS token if your feed requires it. Then deploy it and watch data flow to Velocity.
Use Cases
gRPC can be particularly useful in cases where you need to fetch data from an API with particular query parameter requirements, pagination methods, or that produce highly nested data structures. For example, Dataminr and Factal provide threat data streams via APIs and a custom gRPC client can be implemented with logic to page through query results and transform the data into a form that works for a customer's use case.
Other good use cases would be message brokers such as Kafka or MQTT with security implementations that are not yet out-of-the-box with ArcGIS feeds. An organization could develop a custom app that would connect to the broker and receive data which it would then package into gRPC messages and send them to Velocity.
Resources
The new capability comes with online documentation that explains how to configure gRPC feeds in the Velocity UI. Additionally, because gRPC is inherently developer-focused, we are also providing resources for developers in the form of documentation and reference implementations that illustrate how to build gRPC clients for Velocity.
Developer Support
Resources to support the implementation of gRPC clients for Velocity gRPC feeds are found in the realtime-grpc-feed Github repository. The repository includes:
- The velocity_grpc.proto file which developers will use to generate client classes compatible with Velocity gRPC feed services
- Developer documentation
- Pre-compiled boilerplate code for several languages (these are the result of compiling the velocity_grpc.proto file for each language using the Protocol Compiler)
- Sample reference implementations that can be cloned and leveraged in your own codebase
Summary
With this release developers now have new and extensible means of connecting streaming data to ArcGIS and ingesting it efficiently. This especially supports cases where custom clients need to send data directly to Velocity faster than standard HTTP RESTful requests normally support. And organizations developing applications in multiple languages or deploying clients in CPU or bandwidth constrained environments can also leverage this lightweight option.
See our Resources page for more information about ArcGIS Velocity, including product videos, lessons, documentation and more.