Latest Contributions by PeterNasuti

In this blog we will cover how to use ArcGIS Velocity to change data geometry as records are processed in real-time for a given track based on prior locations. Let’s say that you are responsible for monitoring entities moving in real-time such as vehicles or mobile devices. Most of the time, the entity location is available in attributes X/Y pairs of latitude and longitude or a single geometry field. ArcGIS Velocity converts these location values into point geometry as part of data ingestion for a feed or source. Sometimes we hear a requirement from Velocity users that instead of just knowing the current location of that observation, there is a need to also represent where that entity had been in its previous observations. This is valuable for understanding movement, directionality, and to support advanced processing and decision-making. We will review in depth the Velocity tools, track function and Arcade expression configuration that can be used to achieve this goal. What is ArcGIS Velocity? ArcGIS Velocity is the real-time and big data processing and analysis capability of ArcGIS Online. It allows you to import, visualize, analyze, store, and use data from Internet of Things (IoT) sensors. For general information on what ArcGIS Velocity can do, please visit the product page and product documentation. In real-time analysis, tracks are sequentially ordered features (by the Start Time field) with a field specified as a track identifier (Track ID key field). For example, if you were monitoring fleet vehicles moving around, the Track ID could be a field like the Vehicle ID or the License Plate or the Driver ID. The features in the track would be ordered by their date of arrival to the real-time analytic. Getting started To perform track history analysis for unique moving entities (i.e. unique vehicles, planes, or mobile devices), a user would need to start by configuring a feed in ArcGIS Velocity. As a requirement to use track functions in this workflow is ensuring that the feed has three things present to support this analysis case: Point geometry Start time Track ID field All these components would be identified on “step 3 – Identify Key Fields” of feed configuration. To follow a guided tutorial on feed configuration and setting these properties, visit this tutorial. Once the feed is created, it would need to be added to a real-time analytic. This can be achieved from the feed details page button “Add to new analytic”, or by opening a real-time analytic and adding a feed from the “Add Node” toolbar. Configuring the analytic Now that your feed exists to bring in real-time data and you have added it to a real-time analytic, we can configure the tools and Arcade expression to perform our analysis. We will change the geometry of each record using an Arcade expression. This can be accomplished using either the Calculate Fields or Map Fields tool connected to your feed or the output of another real-time tool performing analysis upstream. Step 1: Add the Calculate Fields tool and connect it to your analytic Click the “+” button on the left pane of the ArcGIS Velocity real-time analytic, and then expand the “Manage Data” folder, and click the “Calculate Field” tool. This will add a disconnected “Calculate Field” tool present on your analytic canvas. Drag the tool to the desired location, and then drag the output port from a feed or upstream tool in your analytic to connect to the target port of the Calculate Fields tool. Step 2: Configure the Calculate Fields tool to add a geometry expression Now that the Calculate Fields tool is connected, double click (or right click > view properties) the tool to open the tool properties pane on the right. Next, click “Add field calculation” and select “Geometry”. This will open the geometry configuration where you can control spatial reference and provide the Arcade expression. Click “</> Configure an Arcade expression” to open the Arcade interface. Step 3: Provide an Arcade expression Within this Arcade expression environment, we will be providing the below Arcade expression that converts point geometry of an observation into a polyline geometry. All the other attributes for the feature will remain the same. The vertices of the polyline geometry are made up of the current and prior two observations for this track ID. Paste this expression into the Calculate Fields “Configure an Arcade expression” window.     // Part 1: Use the TrackGeometryWindow function to return an array // of the prior feature for the track geometry and the // current feature for the track geometry var featureGeometryArray = TrackGeometryWindow(-2, 1) // Part 2: Calculate the number of records in the array var numFeatures = Count(featureGeometryArray) // If this is processing the first or second // observation for a track return null geometry // Part 3: if (numFeatures < 3) { return null; // Part 4: } else { // When there are two prior observations present:, // create polyline geometry between the current // observation and prior observations for this Track ID return Polyline({ "hasM": false, "hasZ": false, "paths": [[[featureGeometryArray[0].X,featureGeometryArray[0].Y],[featureGeometryArray[1].X,featureGeometryArray[1].Y],[featureGeometryArray[2].X,featureGeometryArray[2].Y]]], "spatialReference": {"wkid": 4326} }); }       Let’s review the four key parts of the provided sample expression: Arcade expression part 1: Use the TrackGeometryWindow function First, on line 4 we declare a variable called “featureGeometryArray”. This variable is assigned to the output of the TrackGeometryWindow() function. This function returns an array that includes geometries for the specified range of features. Note that the parameters supplied to the TrackGeometryWindow function in this case are -2 and 1, but can be adjusted as needed in some examples shown below. In this case, -2 is the startIndex and 1 is the endIndex value, and these control how many feature geometries should be returned by the TrackGeometryWindow function. The current feature being processed is index 0. Therefore the startIndex value (first parameter) of -2 tells the function to get the two feature geometries prior to the current feature. The endIndex value or 1 is the feature at the end of the window which will not be returned, which means feature 0 (the current feature) will be returned. Some more examples of parameter values that could be provided to this function: Return prior feature and current feature: (-1,1) Return prior two features and current feature: (-2,1) Return prior three features and current feature: (-3,1) Return prior feature only and not current feature: (-1,0) Arcade expression part 2: Count the length of the array returned by TrackGeometryWindow function In part 1 of this expression, we return an array from the TrackGeometryWindow function. This array includes point feature geometries for the range of features specified by the parameters supplied to TrackGeometryWindow. In the second part (line 7), we declare a variable called numFeatures which uses the Count() function on the featureGeometryArray to understand how many feature geometries are present in the array. When a real-time analytic processes the first observation for a given Track ID, the TrackGeometryWindow function won’t be able to return the prior observation(s) requested by the TrackGeometryWindow parameters. Therefore, we count the length of this array and store it to a variable to understand when a given Track ID is ready for processing. Arcade expression part 3: Return null geometry for the first record(s) of a Track ID On lines 12 and 13, we start our if/else conditional handling with Arcade. In this case, if the number of feature geometries returned (numFeatures) in the array created by the TrackGeometryWindow is less than three, we return a null geometry. This is to handle the case of the first or second observation for a Track ID in real-time processing when there are not prior features. We choose a value of three because we are asking for two prior features in this TrackGeometryWindow function, plus our current feature. If we were asking for four prior features (TrackGeometryWindow(-4,1)) we would specify a value in line 12 of numFeatures < 5. We will cover later in this blog how to use the Filter by Expression tool to filter out and remove these features with null geometries from a processing pipeline if you do not want to retain them. Arcade expression part 4: Return polyline geometry once prior observations are present Lines 15 – 26 starts with handling of the “else” condition, and then simply returns a polyline object using the Polyline() Arcade function. The key component to this part is line 23, where in the polyline object that we are returning we supply X/Y point geometry values as the vertices that make up our polyline. Consider section “featureGeometryArray[0].X”. This tells Arcade evaluation to go to the featureGeometryArray that contains the geometries of prior features. This pulls the first item from this array using array slicing of [0] immediately after the array. The first item from the array is the oldest feature geometry relative to our current feature. This is because the TrackGeometryWindow function starts processing and adding features to the array from the specified startIndex. And finally, we call “.X” to pull the X coordinate from this geometry object. The next part uses “.Y” to pull the Y coordinate from the geometry object. Because we configured lines 4 and 12 to return the two prior features and the current feature (3 total), we specify three vertices constructed with X/Y value replacement: [featureGeometryArray[0].X,featureGeometryArray[0].Y] is the oldest feature geometry for the Track ID [featureGeometryArray[1].X,featureGeometryArray[1].Y] is the prior feature geometry for the Track ID [featureGeometryArray[2].X,featureGeometryArray[2].Y] is the current feature geometry for the Track ID If you specified more or fewer features to be returned on line 4 and 12, you will need to adjust the number of vertices specified on line 23. Next steps Once you have provided this Arcade expression, click “Run” in the top left to test and evaluate the expression. If everything is configured correctly, the output will be “geometry” type of a polyline. You can then click “Ok” to return to the Calculate Fields tool, click “Add field calculation”, and then “Apply” to save your changes to the tool. If you click the schema tab on the right side of the node properties pane, you can now see that this Calculate Fields tool is generating polyline geometry. (Optional) Filtering out features with null geometry Recall that on line 13 of our Arcade expression, if there weren’t enough features present for a given Track ID to generate our polyline geometry, we simply returned a null geometry. Many organizations would want to discard these features to ensure that there is not a mix of features with and without geometry causing confusion in later tool processing, analysis, or application experiences. To get rid of these features, add a “Filter by Expression” tool after your Calculate Fields tool. Configure an Arcade expression such as below, where only features that have a geometry that is not null are retained. The expressions provided to the Filter by Expression tool need to result to a Boolean: true or false, and the features which evaluate in the expression to true are passed to the next step of analysis.     Geometry($feature) != null     Review output results Let’s take a quick look at the result. Originally the real-time data was point geometry for each track representing city buses in the Washington DC region. For additional context, green arrows show the buses direction of travel. Note that this feature observation DateTime value is at 4:18:33 AM Next, we can click on a polyline which was generated by our Arcade expression. It has the same attributes as the vehicle point above, however the geometry was changed to a polyline. In this polyline the vertices are created in order from the older historical observations for the track towards the more recent observations. Conclusion This is just one example of the many advanced analytic opportunities made available through the concept of Arcade Track Functions within ArcGIS Velocity field and geometry calculation tools. These track functions provide the ability to “look backwards” in time for a given unique set of observations grouped by Track ID and ordered by the feature Start Time timestamp value. These functions, combined with the many other Arcade functions available provide an extensibility story where through analytic and tool sequence users can tailor real-time and big data analysis pipelines to meet the specific needs of their organization. There are many more track functions including the below that can be used for a variety of other solutions pertaining to acceleration, distance, speed, duration, attribute field values, and more. From the Velocity product team, we look forward to writing more blog posts highlighting how these can be used to answer common real-time questions. Additional Track Functions https://developers.arcgis.com/arcade/function-reference/track_functions/ TrackAccelerationAt TrackAccelerationWindow TrackCurrentAcceleration TrackCurrentDistance TrackCurrentSpeed TrackCurrentTime TrackDistanceAt TrackDistanceWindow TrackDuration TrackFieldWindow TrackGeometryWindow TrackIndex TrackSpeedAt TrackSpeedWindow TrackStartTime TrackWindow ... View more