Parsing a Multiple Array json input in GeoEvent Processor

[JasonSphar]

Hi,

I originally tried the post at this link https://community.esri.com/community/gis/enterprise-gis/geoevent/blog/2018/07/25/json-data-structure...  in an attempt to take what looks like a json message with more than one array and parse it properly using GeoEvent.  A sample of my file is attached called sample.json.  Looking at the structure it looks like I enter into an array at the third line called "sensors": [    

I then hit what looks like a nested array within the original array (might be using wrong terminology here, sorry) at line 12 called "data":[

Following the article referenced in the link above, I am creating a definition file that looks like the image attached called GeoEventDEF.png.

I am new to using json files in GeoEvent and was wondering if the definition file show in the image attached is indeed correct for the attached sample json file?  

Additionally, I was also curious that if a suggested solution is proposed that works, what is the best way to move this data into a feature service?  I ultimately want to flatten the data into a table where all key value pairs have their respective field and value in a backend database table or feature class.

Any help would be greatly appreciated.  

Thanks, Jason

[DanielCota1]

Hi Jason Sphar‌,

There a couple of things I would like to address...

The definition looks like what would be expected from the sample JSON you provided. It creates a couple of grouped fields where the nested JSON is detected. That is how Geoevent parses it based off the JSON structure.

However, depending on your desired output, you may want to configure the input connector to only pull from a certain subset of the JSON. For example, if you only want the data under "data' to be parsed and processed through Geoevent, you will want to set "data" as the JSON Object Name in the input connector configuration.

If you want ALL of the data in the JSON to go through, you will want to manually create a flattened definition that will then be used for the feature service output. From there, you can use a field mapper in the Geoevent Service to map the hierarchical definition from the input to the flattened definition on the output. When creating the flattened definition, you will want to pay close attention to the schema and the field types that the definition is using. It should match the original as much as possible.

One last point to consider, and this may not hold true for your final workflows, but the sample JSON you provided does not seem to include any spatial data that can be used to construct geometry for a feature service, at least not from what I can tell. If you want to add this data to a feature service that can be displayed on a web map or other mapping client, it needs that spatial component. Now, you may plan on using different data or using a processor to append spatial data within the Geoevent service, but as it stands now, there is none. Just wanted to make sure you were aware.

I hope this helps.

-Daniel

[Stefan_Jung]

Hey Jason Sphar‌,

maybe this custom processor could help to solve our problem:

GitHub - Esri/multi-cardinal-field-splitter-for-geoevent: ArcGIS GeoEvent Server sample Multicardina... 

Setting data as Field to be splittet as multiple GeoEvents would result following text Output that could also be saved to a relational database:

3,Application Specific Float,10,12,100665857,4088,Generic Table,2018-10-05T00:26:00.000+02:00,328,true,0
3,Application Specific Float,10,12,100665857,4088,Generic Table,2018-10-05T00:26:00.000+02:00,81.4000015258789,true,3
3,Application Specific Float,10,12,100665857,4088,Generic Table,2018-10-05T00:26:00.000+02:00,0.1979999989271164,true,2
3,Application Specific Float,10,12,100665857,4088,Generic Table,2018-10-05T00:26:00.000+02:00,0.17100000381469727,true,4
3,Application Specific Float,10,12,100665857,4088,Generic Table,2018-10-05T00:26:00.000+02:00,0.0820000022649765,true,5
3,Application Specific Float,10,12,100665857,4088,Generic Table,2018-10-05T00:26:00.000+02:00,0.1940000057220459,true,1

Best,

Stefan

[JasonSphar]

Thanks for the replies. We have a way to get the data into a csv file that is stacked and not flat, meaning I now have a field named “value” that stores only one number but that number, and the ones after it, need to match fields in a backend feature service that are flat.  I have tried to filter the data based on another field we have called “sensor tag” that for the most part is unique and then use that filter to map the number in the “value” field to its respective field in the backend feature service. The problem is that after filtering the data then using it in the field mapper, the only field that makes any sense to use is the original “value” field but it doesn’t seem that value actually changes for each individual filter and so when I field map it I’m  unable to get each unique number that I need to parse to the proper field on the back end. If the data was not stacked and had unique  field names this wouldn’t be so tough but they are. Any advice on good ways to move stacked attributes to flat with GeoEvent?

Thanks, Jason 

[JeffreyWilkerson]

Jason,

This looks like the same request that I responded to at https://community.esri.com/message/838354-re-parse-json-location-data-from-array?commentID=838354#co....  Your given a JSON object that has multiple entries that you need to parse through in order to feed them into GeoEvent server one at a time.  The current JSON connector in GeoEvent really only works on one entry at a time from what I can tell.  I think a generic connector could be developed that would look for one, or maybe even 2 levels of multiple entries within one JSON object, but that's something for another time. 

The Python code that was shown in the last link could be used in this instance as well by cycling through all 'sensors' (with 'for sensor in output['sensors']') and then pulling each individual 'data' value through another loop (i.e. 'for dataValue in sensor["data"]).  You may only need the second loop if 'sensors' could only occur once in the JSON feed. 

The problem I see with the values in data is there is no uniqueness other than the value field.  The time stamp, encoding, byte_array, and is_number is the same for each entry.  I would think you would want to register these values to something specific so you could push them out in the order they are showing, like 'Value0', 'Value1', etc.  If they mean something in order then I would use that (i.e. 'Azimuth', 'WindSpeed', etc.) in your redefined stream.

I don't think running Python to 'clean' the JSON input and then pushing to a TCP port is the long term solution, but it's all I have up now.  When I get a chance I am hoping to do the same thing in a Java based, GeoEvent input connector, hopefully, someday, unless someone else beats me to it.

Good luck, Jeff.