Latest Contributions by BrianWatsonEsri

ArcGIS Velocity enables users to connect to and ingest data from a wide variety of data sources and data formats.  One of those data formats is the incredibly popular JSON (JavaScript Object Notation).  JSON is a flexible format that is programming language independent, lending to its popularity.  Sometimes challenges are encountered due to this flexibility.  For one, JSON data can be formatted and arranged in many ways. ArcGIS Velocity aides resolving this challenge by not only ingesting the JSON, but also transforming it into a common format—an ArcGIS Feature. At a basic level, an ArcGIS Feature contains geometry and attribute information. It is possible to have features with only one or the other, however most often in practice we see users working with spatial data with some sort of attribute information so a feature will contain both geometry and attributes. The job of ArcGIS Velocity is not only to connect to various data sources, but to also parse and map that data to the appropriate geometry and attributes to properly construct an ArcGIS Feature. In this article we will first review some basics of JSON data.  We will then discuss how to work with basic JSON data types in ArcGIS Velocity and then follow up with working with more complex JSON data configurations. Basics of JSON Before diving into the conversion of JSON to Features in ArcGIS Velocity, it might be helpful to review some JSON basics.  If you are already familiar with JSON, feel free to skip ahead to the next section. Primary JSON structure data types JSON provides two key data types for structuring data: the object, and the array. Object:  a collection of key-value pairs, delimited by curly brackets.  The keys, often referred to as “fields”, serve as string labels and values can be any JSON data type such as a string, number, or Boolean for example.       { "type": "truck", "speed": 65.342, "color": "silver", "timestamp": 946684799000 }         Array:  an ordered list of zero or more values of other JSON data types, delimited by square brackets.       [ "4b1802", "SWR805G ", "Switzerland", 1610556596, 8.557, 47.4536, false, 0 ]       Basic JSON data types JSON provides basic data types in addition to the structure types above. Number:  a number that may contain a fractional part or may be a whole number (JSON does not make a distinction in numeric types). String:  a sequence of characters, wrapped in double quotes. Boolean:  either ‘true’ or ‘false’, without any quotes. Null:  represented by ‘null’ without quotes. Signifies an empty value, there is no data for this field. Here’s a small example of these basic data types in an object.       { "number": 23.456, "string": "velocity", "boolean": true, "nullValue": null }        JSON types in combination To keep things interesting (and sometimes challenging!) JSON can be represented as a mixture of the above types. For example, a field in a JSON object can be another object. We often call these ‘nested’ objects.  In the following example, we would call the “iss_position” field a nested object because it is an object nested under the root object that also contains the “timestamp” and “message” fields.       { "timestamp": 1610561686, "message": "success", "iss_position": { "longitude": "-6.9611", "latitude": "-7.9850" } }         An array could have objects or even other arrays as its elements.  The following is an example of an array of objects.       [ { "type": "truck", "speed": 65.342, "color": "silver", "timestamp": 946684799000 }, { "type": "sedan", "speed": 55.496, "color": "black", "timestamp": 946684800000 }, { "type": "hatchback", "speed": 74.198, "color": "red", "timestamp": 946684805000 } ]       Working with basic JSON in ArcGIS Velocity When configuring a feed or data source in ArcGIS Velocity and sampling generic JSON data, ArcGIS Velocity attempts to derive the schema of the data based on the data retrieved from the feed or source type.  There are some basic rules ArcGIS Velocity follows when parsing JSON data. Object If the root of the data is an object, the entire object is treated as one feature.  Any keys within it are considered attributes of that feature.  Below the root level, any nested objects are derived as string fields.  As we will see below, we have options to where we can define what is the root level to modify what ArcGIS Velocity will ingest as a feature. If we take the example of an object with a nested object above and sample that data in ArcGIS Velocity, we get the following where we can see the object, “iss_position” is a string:       { "timestamp": 1610561686, "message": "success", "iss_position": { "longitude": "-6.9611", "latitude": "-7.9850" } }           In our current release, ArcGIS Velocity derives all arrays as string fields. Arrays offer a unique challenge for ingestion given their one-to-many nature and an indeterminate number of elements in them.  There are even more advanced techniques for parsing arrays using Arcade expressions in analytic tools that will be covered in a later article. There is one unique exception to arrays and that is an array of objects.  If the root level is an array of objects with matching keys, each object in the array is treated as a feature, thereby ingesting multiple features from the single array. If we take the example above and sample that in ArcGIS Velocity, we would see the following:       [ { "type": "truck", "speed": 65.342, "color": "silver", "timestamp": 946684799000 }, { "type": "sedan", "speed": 55.496, "color": "black", "timestamp": 946684800000 }, { "type": "hatchback", "speed": 74.198, "color": "red", "timestamp": 946684805000 } ]         JSON array of objects derived as multiple samples. Basic Data Types The remaining basic data types are straight-forward. Number:  Velocity has several unique numeric field types and will derive the type based on the presence of decimal values. String:  JSON strings will be derived as String field types. Boolean: JSON Booleans will be derived as Boolean field types. Null:  If all samples for a key contain null values, Velocity will derive that key as a String field type. Taking the example of basic data types above, we get the following in ArcGIS Velocity:       { "number": 23.456, "string": "velocity", "boolean": true, "nullValue": null }         Basic JSON data types as derived in ArcGIS Velocity. Working with Complex JSON in ArcGIS Velocity As noted above, JSON representing data of interest can and often is presented in a mixture of the above data types.  This flexibility and variety of formatting proves a challenge in parsing the data into features.  ArcGIS Velocity provides some additional tools to work with this complexity. Root Node Sometimes the data of interest is nested within an initial object and the data ‘above’ that JSON field is not of interest.  Defining a root node allows users to define a new root of the JSON data by defining the name of that JSON field.  ArcGIS Velocity will search through the entire JSON dataset until it finds that field.  If the root node defined by the user is an object, any fields within the root node become the new feature attributes.  In the example above with a nested object, we could define the field “iss_position” as the Root node and click to derive the schema again so that “iss_position” now becomes the feature and any keys within it are the new feature attribute fields.       { "timestamp": 1610561686, "message": "success", "iss_position": { "longitude": "-6.9611", "latitude": "-7.9850" } }         Attribute schema of JSON data after defining root node.   If the root node is an array of objects with matching keys, each object becomes a feature.  This is particularly common with many data providers where there is often metadata provided at the root level and the data of interest is often found in array ‘further down’ in the data. Flattening & Flattening Exemptions The root node format property provided a way to define a root node and effectively bypass data above that node and only ingest the nested data.  There might be scenarios where ingesting data at all levels within an object with nested objects is desired.  This is where flattening comes in.  Here is an example of JSON data where you might want to maintain all data:       { "name": "launch vehicle", "alt": 423.89075574805, "nested_object": { "norad_ID": "00001", "velocity": 27593.53333242, "time_string": "2020-01-01T00:00:00.000Z", "geometry": { "x": 63.30655171795018, "y": 45.96504747203276, "spatialReference": { "wkid": 4326 } } } }       Before flattening, Velocity derives the schema as follows: Attribute schema derived from JSON data before flattening. JSON flattening effectively creates a singular object from multiple nested objects.  Any JSON keys nested within lower objects become new fields with their key name appended to the object’s key name delimited by underscores by default.  You can modify these field names to suit your needs. The same rule for arrays applies as above.  An array at the root or under nested objects will derive and be ingested as a string. If we were to check Flatten checkbox and derive the schema, we get a new schema with all fields: Attribute schema after flattening JSON data. JSON flattening is very powerful, however there are times when you might not want all data flattened.  Flattening Exemptions allow you to provide a comma separated list of field names to exempt from flattening.  This is commonly used for avoiding flattening geometry objects that conform to a geometry standard like GeoJSON, EsriJSON, or Well-known text. We can take the example above and add “geometry” to the Flattening exemptions and now see that object preserved as a string: Attribute schema after defining a flattening exemption. It is important to note enabling flattening and defining root nodes can be done in combination providing for additional flexibility and control of the data you are ingesting. In the example below, we can configure JSON data ingestion where we are providing a root node, flattening the schema, and also providing a flattening exemption to maintain a geometry object. If we had the same data used in the flattening examples above, but instead it was in a “data” field of the root object, we could get the same result by defining the root node as “data”.       { "data": { "name": "launch vehicle", "alt": 423.89075574805, "nested_object": { "norad_ID": "00001", "velocity": 27593.53333242, "time_string": "2020-01-01T00:00:00.000Z", "geometry": { "x": 63.30655171795018, "y": 45.96504747203276, "spatialReference": { "wkid": 4326 } } } }, "timestamp": 1610700042 }         Attribute schema with a root node defined, flattening, and a flattening exemption defined.  Conclusion In this article we have provided an overview of the basics of JSON data format and displayed how ArcGIS Velocity provides many tools to easily bring a variety of JSON data configurations into the ArcGIS platform.  You can take any of the JSON examples above and experiment with the authoring workflow in ArcGIS Velocity using the HTTP Receiver feed.  The initial configuration and data sampling interface of the HTTP Receiver feed type provides an area to provide sample data where you can paste JSON data in and then derive the schema.  In a future article, we will outline steps to tap into JSON even further using analytic tools and Arcade Expressions to parse JSON in string fields.   ... View more