Latest Contributions by KenGorton

The new Data Sampler in ArcGIS Velocity lets you inspect event messages from one processing step to the next to verify how it is being transformed within a real-time analytic. ... View more

Most feeds in ArcGIS Velocity connect to real-time data by either subscribing to a remote data source such as Kafka, MQTT, or Azure Event Hub, or polling a data source such as a third-party provider API. ArcGIS can also receive real-time data on a REST endpoint by leveraging the HTTP Receiver feed type.  And in the December update of Velocity, there is a new feed type that can be used to receive data:  gRPC. The gRPC feed type creates a built-in gRPC server endpoint that can be used to receive data sent from remote gRPC client applications. Client applications can be developed in a variety of languages, as supported by the gRPC framework. The gRPC feed type is currently supported in Beta and enables developers to become familiar with implementing gRPC clients for Velocity. gRPC uses a .proto file to define the communication between gRPC clients and services. The .proto file can be compiled in numerous languages to produce native client libraries. Clients can connect to services created from the same .proto file regardless of the languages used. Why gRPC? gRPC is a modern, lightweight, high-performance framework that depends on a contract-first approach to API development. It is language agnostic and allows for implementation in a wide variety of development environments. When to use gRPC? When you need to ingest real-time data into ArcGIS from providers or in data formats that ArcGIS Velocity does not support out-of-the-box:  as with the HTTP Receiver feed type, you can develop a client application to push data into an ArcGIS feed in a supported format When you need to implement custom clients to stream data into ArcGIS: gRPC supports both streaming and unary calls, while pushing data in over an HTTP endpoint only supports unary calls. When you have environments with clients developed in diverse languages and all need to connect to ArcGIS Velocity. When efficient communication is paramount such as in CPU-constrained or limited bandwidth environments. gRPC in Velocity The most common approach for implementing a gRPC-based ingestion solution will be to first configure a gRPC feed in Velocity. The resulting feed details page will provide essential connection information to be leveraged when creating and deploying client applications. Configure a gRPC feed much like you would other feed types. It is similar to the HTTP Receiver feed type in that you must supply sample data from which to derive a schema or manually build the schema field-by-field. The feed details page will provide three key values that you will need when developing gRPC clients: the gRPC endpoint URL, gRPC endpoint header path and whether the feed is configured with ArcGIS OAuth 2.0 authentication. With these connection parameters, you will select the development environment of your choice and write the client code to connect to your gRPC feed and send it data. That development effort will require the velocity_grpc.proto file. The .proto file is a key element of a gRPC implementation in that it defines the messages that gRPC clients and services may exchange with one another and serves as the contract under which they communicate. This file must be compiled for your language of choice to generate the client code classes. You can compile it with Google’s Protocol Compiler, but some integrated development environments will compile it when you build your code. When your client application is ready you will provision it with the . Don’t forget to configure your client to use an ArcGIS token if your feed requires it. Then deploy it and watch data flow to Velocity.   Use Cases gRPC can be particularly useful in cases where you need to fetch data from an API with particular query parameter requirements, pagination methods, or that produce highly nested data structures. For example, Dataminr and Factal provide threat data streams via APIs and a custom gRPC client can be implemented with logic to page through query results and transform the data into a form that works for a customer's use case. Other good use cases would be message brokers such as Kafka or MQTT with security implementations that are not yet out-of-the-box with ArcGIS feeds. An organization could develop a custom app that would connect to the broker and receive data which it would then package into gRPC messages and send them to Velocity.   Resources The new capability comes with online documentation that explains how to configure gRPC feeds in the Velocity UI. Additionally, because gRPC is inherently developer-focused, we are also providing resources for developers in the form of documentation and reference implementations that illustrate how to build gRPC clients for Velocity.   Developer Support Resources to support the implementation of gRPC clients for Velocity gRPC feeds are found in the realtime-grpc-feed Github repository. The repository includes: The velocity_grpc.proto file which developers will use to generate client classes compatible with Velocity gRPC feed services Developer documentation Pre-compiled boilerplate code for several languages (these are the result of compiling the velocity_grpc.proto file for each language using the Protocol Compiler) Sample reference implementations that can be cloned and leveraged in your own codebase   Summary With this release developers now have new and extensible means of connecting streaming data to ArcGIS and ingesting it efficiently. This especially supports cases where custom clients need to send data directly to Velocity faster than standard HTTP RESTful requests normally support. And organizations developing applications in multiple languages or deploying clients in CPU or bandwidth constrained environments can also leverage this lightweight option.   See our Resources page for more information about ArcGIS Velocity, including product videos, lessons, documentation and more.   ... View more

  Why HTTP Receiver Most feeds in ArcGIS Velocity receive data by either subscribing to a remote data source such as a Kafka broker, MQTT or Azure Event Hub, or polling a data source such as an ArcGIS feature layer or an external website. But there are many platforms and systems that don’t accept connections, or allow polling of their resources. Many of these systems instead have the capability to send data to other platforms over HTTP. For these cases, you can use the Velocity HTTP Receiver feed type. The HTTP Receiver feed type allows you to enable a REST endpoint hosted by ArcGIS, to which external platforms, apps and APIs can submit POST requests containing payloads of data. Velocity doesn’t subscribe to or poll the external source at regular intervals, it listens on the REST endpoint for data. Configuring an HTTP Receiver feed The steps to setting up the HTTP Receiver are very similar to configuring any other feed. Get started by clicking Create feed on the Velocity home page or Feeds page. The HTTP Receiver feed in Velocity is found in the Web and Messaging feed types group under HTTP.         Next, configure whether to require authentication. HTTP Receiver enables you to configure the feed to require authentication in the form of an ArcGIS token. In this case, each POST request sent to the feed’s HTTP endpoint  must include a valid token or it will be rejected. Alternatively, you can require no authentication.   On the Provide Schema step, Velocity usually samples data from the source and automatically parses it to predict the schema.  Because the REST endpoint does not yet exist, data cannot be automatically sampled.  Instead, you have two options. You can manually configure the expected fields, which is easier if you have simpler data such as delimited text. Or if instead you have something more complex like nested JSON or XML, our recommended practice is to provide a few samples of data to Velocity for schema prediction. This further facilitates additional formatting steps like flattening or specifying the root node.   The remaining steps are the same as with other feeds: you identify the key fields containing location information, date/time values and the Track ID, assign a name and description and save the feed. The resulting feed has a key property: the HTTP endpoint path. This is the URL you will provision to your external system in order to direct its POST requests to ArcGIS Velocity.   Provisioning the App or API There are countless examples of apps and APIs that can send data via POST requests to remote destinations. Many of these may have a tailored user interface where you can configure the destination URL to which the app should send data. A great example of this is Garmin inReach, a two-way satellite-based tracking and messaging device from Garmin. An organization managing multiple inReach devices for their field staff uses Garmin’s Portal Connect website to administer them. Portal Connect enables an administrator to configure an outbound URL so that all the track points and messages from each inReach device will automatically be forwarded to a destination such as Velocity. In Portal Connect simply paste the HTTP endpoint path URL of the feed, hit Test and Save.   Other apps may enable a developer to build the REST requests into the app’s code or configuration files to enable the app to send data to Velocity. To test this, you can even use an HTTP Post utility such as Postman. These allow developers to send HTTP requests to test different configurations with a great deal of flexibility. In Postman: 1.       Select POST for the request type. 2.       Paste the HTTP endpoint path value into the address bar. 3.       In the Body tab, choose the raw encoding type and paste a sample of the data to be sent. This should be in the same format and schema as the sample you used to create the feed. 4.       Click Send. The response will appear near the bottom of the screen.       If you configured your feed to require ArcGIS authentication, you must first obtain a valid token. A method of doing so from Postman is to: 1.       Add a new tab so that you keep the request configured above.           2.       Select POST for the request type 3.       Enter https://www.arcgis.com/sharing/rest/generateToken in the address bar.   4.       On the Params tab enter the following keys and values: a.       referer: https://www.arcgis.com/ b.       f: pjson     5.       On the Body tab select x-www-form-urlencoded and enter the following keys and values: a.       username: your ArcGIS username b.       password: your ArcGIS password     6.       Click Send. A token will be returned and displayed near the bottom of the screen. Copy the value of the token to your system clipboard. Do not include the quotation marks.   Then in the tab where you configured the request to send data to your HTTP Receiver feed go to the Authorization tab, select Bearer Token in the Type drop-down menu and paste the token in the Token field before hitting Send.   During actual app development, when creating your POST request programmatically, the token must be in the request headers as: Authorization: Bearer <your token>. Refer to the ArcGIS Developer documentation for more about obtaining a token. Refer to the Velocity documentation for more details on the HTTP Receiver feed type. We hope these examples are helpful in getting started with enabling your apps and systems to send real-time data to ArcGIS Velocity. In the comments, let us know what kinds of data streams and systems you are working with today!   ... View more

See how ArcGIS Velocity offers different options for saving real-time data to feature layers. ... View more

Hi GeoNet When I create a new user using the Python API and set the role parameter to a custom role that I know to exist in the org, the resulting user does not belong to the role I specified. If I set the role parameter to be a default role the user ends up in that role. Is there a particular way to set new users into a custom role or is this a known issue? Thanks, Ken ... View more

I recently did a search for tools that will convert feature layers to GPX format. Not finding anything, I ended up writing one. I deployed it to an Azure portal as a severless compute function. You can see the results in this storymap: The 2020 BMW RA Rally  The routes in this storymap were created using the Directions capability in the standard ArcGIS Online web map viewer. The resulting route feature collections were published as hosted feature layers and shared publicly. The URLs to those feature layers are configured as parameters in the links in each of the 'Download GPX file' buttons in the storymap. If anyone is interested you can leverage a similar capability in your own storymaps or other applications using this function: https://arcgis2gpx.azurewebsites.net/api/Layers2GPX It takes GET or POST requests and accepts point or polyline feature layers. The features in point feature layers become GPX waypoints while the features in polyline feature layers become GPX tracks. The minimum parameters required for GET requests is the url to a point or polyline feature layer. Thus you can create a URL for a hyperlink in a web page, storymap or other document as follows:  https://arcgis2gpx.azurewebsites.net/api/Layers2GPX?url=https://services1.arcgis.com/EvDRLcHhbHG5BnwT/arcgis/rest/services/Hiking_Routes/FeatureServer/0&name=Name&desc=Description or https://arcgis2gpx.azurewebsites.net/api/Layers2GPX?url=https://services1.arcgis.com/fBc8EJBxQRMcHlei/arcgis/rest/services/Appalachian_National_Scenic_Trail/FeatureServer/0  The minimum payload for a POST request is a json object with a title element (which will become the output GPX filename) and an array of feature layer urls. Thus you can combine multiple point and/or polyline feature layers into a single GPX file. Example: { "title":"Some Trails", "returnFile":"true", "layers":[ { "url":"https://services1.arcgis.com/EvDRLcHhbHG5BnwT/arcgis/rest/services/Hiking_Routes/FeatureServer/0", "name":"Name", "desc":"Description", "where":"AscentFT>3600" }, { "url":"https://services1.arcgis.com/fBc8EJBxQRMcHlei/arcgis/rest/services/Appalachian_National_Scenic_Trail/FeatureServer/0", "name":"Name", "desc":"Description", "where":"Length_Ft>40000" } ] } Additionally, both methods allow you to include optional parameters for each url to indicate wich fields from the source feature layers to utilize for the different GPX elements such as name, desc, cmt, etc Click the function URL above for a complete list of optional parameters. I cannot guarantee how long the function will remain up. If you want the code let me know. If you find an error in functinality or behavior, please also let me know. Ken ... View more