ArcGIS Velocity – Receiving data from custom apps or third party solutions using the HTTP Receiver feed

[#1]

Why HTTP Receiver

Most feeds in ArcGIS Velocity receive data by either subscribing to a remote data source such as a Kafka broker, MQTT or Azure Event Hub, or polling a data source such as an ArcGIS feature layer or an external website. But there are many platforms and systems that don’t accept connections, or allow polling of their resources. Many of these systems instead have the capability to send data to other platforms over HTTP. For these cases, you can use the Velocity HTTP Receiver feed type.

The HTTP Receiver feed type allows you to enable a REST endpoint hosted by ArcGIS, to which external platforms, apps and APIs can submit POST requests containing payloads of data. Velocity doesn’t subscribe to or poll the external source at regular intervals, it listens on the REST endpoint for data.

Configuring an HTTP Receiver feed

The steps to setting up the HTTP Receiver are very similar to configuring any other feed. Get started by clicking Create feed on the Velocity home page or Feeds page. The HTTP Receiver feed in Velocity is found in the Web and Messaging feed types group under HTTP.

Next, configure whether to require authentication. HTTP Receiver enables you to configure the feed to require authentication in the form of an ArcGIS token. In this case, each POST request sent to the feed’s HTTP endpoint  must include a valid token or it will be rejected. Alternatively, you can require no authentication.

On the Provide Schema step, Velocity usually samples data from the source and automatically parses it to predict the schema.  Because the REST endpoint does not yet exist, data cannot be automatically sampled.  Instead, you have two options. You can manually configure the expected fields, which is easier if you have simpler data such as delimited text. Or if instead you have something more complex like nested JSON or XML, our recommended practice is to provide a few samples of data to Velocity for schema prediction. This further facilitates additional formatting steps like flattening or specifying the root node.

The remaining steps are the same as with other feeds: you identify the key fields containing location information, date/time values and the Track ID, assign a name and description and save the feed.

The resulting feed has a key property: the HTTP endpoint path. This is the URL you will provision to your external system in order to direct its POST requests to ArcGIS Velocity.

Provisioning the App or API

There are countless examples of apps and APIs that can send data via POST requests to remote destinations. Many of these may have a tailored user interface where you can configure the destination URL to which the app should send data.

A great example of this is Garmin inReach, a two-way satellite-based tracking and messaging device from Garmin. An organization managing multiple inReach devices for their field staff uses Garmin’s Portal Connect website to administer them. Portal Connect enables an administrator to configure an outbound URL so that all the track points and messages from each inReach device will automatically be forwarded to a destination such as Velocity. In Portal Connect simply paste the HTTP endpoint path URL of the feed, hit Test and Save.

Other apps may enable a developer to build the REST requests into the app’s code or configuration files to enable the app to send data to Velocity. To test this, you can even use an HTTP Post utility such as Postman. These allow developers to send HTTP requests to test different configurations with a great deal of flexibility. In Postman:

1.       Select POST for the request type.

2.       Paste the HTTP endpoint path value into the address bar.

3.       In the Body tab, choose the raw encoding type and paste a sample of the data to be sent. This should be in the same format and schema as the sample you used to create the feed.

4.       Click Send.

The response will appear near the bottom of the screen.

If you configured your feed to require ArcGIS authentication, you must first obtain a valid token.

A method of doing so from Postman is to:

1.       Add a new tab so that you keep the request configured above.      

2.       Select POST for the request type

3.       Enter https://www.arcgis.com/sharing/rest/generateToken in the address bar.

4.       On the Params tab enter the following keys and values:

a.       referer: https://www.arcgis.com/

b.       f: pjson

5.       On the Body tab select x-www-form-urlencoded and enter the following keys and values:

a.       username: your ArcGIS username

b.       password: your ArcGIS password

6.       Click Send.

A token will be returned and displayed near the bottom of the screen. Copy the value of the token to your system clipboard. Do not include the quotation marks.

Then in the tab where you configured the request to send data to your HTTP Receiver feed go to the Authorization tab, select Bearer Token in the Type drop-down menu and paste the token in the Token field before hitting Send.

During actual app development, when creating your POST request programmatically, the token must be in the request headers as:

Authorization: Bearer <your token>.

Refer to the ArcGIS Developer documentation for more about obtaining a token.

Refer to the Velocity documentation for more details on the HTTP Receiver feed type.

We hope these examples are helpful in getting started with enabling your apps and systems to send real-time data to ArcGIS Velocity. In the comments, let us know what kinds of data streams and systems you are working with today!