topic GeoEvent Field Mapper - from flat to multicardinality in ArcGIS GeoEvent Server Questions

GeoEvent Field Mapper - from flat to multicardinality

SpatialSean — Thu, 02 May 2024 02:18:26 GMT

I have a flat dataset that needs to be shifted into multicardinality for the output but am struggling with what syntax needs to be used within the field mapper to go from the 1 to many.

I need to take my fields such as language, event, urgency, and effective and then map them to be included into "info" - It may take multiple sets of field mappings to get it correct.

I've gotten as far as knowing there needs to be a '.' in the field mapper else it fails. For example trying to push the string 'Weather' as a category which is also a many.

[ Edited: RJ Sunderman 5/1/2024 7:11 pm ]

What @SpatialSean wants to do is (practically) impossible.
A custom processor is the best approach for this. Refer to my comment below.

{ "identifier": "123456789", "status": "Actual", "msgType": "Alert", "scope": "Public", "sent": "2019-05-28T13:49:09+0000", "info": [ { "language": "en", "event": "Test Basic Custom Feed Event", "urgency": "Immediate", "severity": "Minor", "certainty": "Observed", "description": "This is a sample description", "headline": "Sample Headline", "web": "http://www.everbridge.com/", "effective": "2019-05-28T13:49:08+0000", "category": [ "Transportation", "Local Disaster" ], "area": [ { "areaDesc": "25 Corporate Drive Burlington, MA", "polygon": [ "-71.07593,42.50798 -71.20801,42.59766 -71.30529,42.55079 -71.31828,42.52278 -71.07593,42.50798" ] } ] } ] }

"language": "null", "event": "Late March Storm System for Plains and Midwest - Snow 3", "urgency": "null", "severity": "null", "certainty": "null", "effective": "Sat Mar 23 07:00:00 EDT 2024", "expires": "Wed Mar 27 19:00:00 EDT 2024",

Re: GeoEvent Field Mapper - from flat to multicardinality

Gene_Sipes — Fri, 29 Mar 2024 13:37:30 GMT

I can give you an example of what mine looks like. Here is a snip from my JSON feed. You can see that the rental uris property is multi-cardinal with a link for iOS and Android.

So... in the mapper it looks like this when trying to specify that I want to map the iOS link to the rental_ios feild in my feature service, as well as the android link.

Re: GeoEvent Field Mapper - from flat to multicardinality

SpatialSean — Mon, 01 Apr 2024 18:03:35 GMT

Thanks Gene. Maybe I was too many steps into it

Here is how my data is leaving the field mapper currently:

{ "GED_Name": "Risk-IN", "identifier": "43311", "sent": "Mon Apr 01 13:40:24 EDT 2024", "status": "Actual", "msgType": "Alert", "category": "null", "headline": "A bunch of info html coded text for detailsRain Threat:     Range, 2-4 in         Impacts: High water, Washouts, Road closures, River flooding", "description": "Another really long block of text", "scope": "null", "language": "null", "event": "Early-Week Severe Weather for the Central U.S. - Rain 1", "urgency": "null", "severity": "null", "certainty": "null", "effective": "Sun Mar 31 19:00:00 EDT 2024", "expires": "Wed Apr 03 07:00:00 EDT 2024", "web": "null", "resourceDesc": "null", "uri": "null", "areaDesc": "Early-Week Severe Weather for the Central U.S.", "polygon": { "rings": [ [ [ -76.96959699999996, 39.28723100000008 ], [ Removing to make this easier to read ] ] ], "spatialReference": { "wkid": 4326 } }, "area": "null" }

I want Category to be this since it is multi cardinal

{ "GED_Name": "Risk-IN", "identifier": "43311", "sent": "Mon Apr 01 13:40:24 EDT 2024", "status": "Actual", "msgType": "Alert", "category": ["Weather"], "headline": "A bunch of info html coded text for detailsRain Threat:     Range, 2-4 in         Impacts: High water, Washouts, Road closures, River flooding", "description": "Another really long block of text", "scope": "null", "language": "null", "event": "Early-Week Severe Weather for the Central U.S. - Rain 1", "urgency": "null", "severity": "null", "certainty": "null", "effective": "Sun Mar 31 19:00:00 EDT 2024", "expires": "Wed Apr 03 07:00:00 EDT 2024", "web": "null", "resourceDesc": "null", "uri": "null", "areaDesc": "Early-Week Severe Weather for the Central U.S.", "polygon": { "rings": [ [ [ -76.96959699999996, 39.28723100000008 ], [ Removing to make this easier to read ] ] ], "spatialReference": { "wkid": 4326 } }, "area": "null" }

I am unsure how in field mapper what syntax is needed to get it at an array:

Re: GeoEvent Field Mapper - from flat to multicardinality

Gene_Sipes — Mon, 01 Apr 2024 18:14:36 GMT

Without seeing your entire setup (mappers, calculators, etc) it's hard to tell what you are trying to do. From these two screenshots, your source fields do not match the incoming json feed properties. For example, you show a source field called DISCUSSION that has a substring method called on it. Where does that come from?

Re: GeoEvent Field Mapper - from flat to multicardinality

SpatialSean — Mon, 01 Apr 2024 18:25:28 GMT

Cutting down from the data in to only show the first 10,000 or 50,000 characters.

On my Target I have a field category that is a string and multicardinality. I want to push the word 'Weather' into it but on the field mapper I am unsure what that syntax is.

Currently I have it written as . + 'Weather' on the field mapper which outputs as:

  "category": "null",

What I want it to result in is:

 "category": ["Weather"],

Re: GeoEvent Field Mapper - from flat to multicardinality

Gene_Sipes — Wed, 03 Apr 2024 15:19:11 GMT

Try something like:
category[0] = "Weather"

Sorry, that's all I got at the moment.

Re: GeoEvent Field Mapper - from flat to multicardinality

SpatialSean — Wed, 03 Apr 2024 19:43:37 GMT

Thank you for all the ideas - no luck. The furthest I got was using a field calculator and attempting the function array('Weather'), however, the array function is not available. Might have to create a custom processor

Re: GeoEvent Field Mapper - from flat to multicardinality

RJSunderman — Thu, 02 May 2024 02:08:31 GMT

@SpatialSean -- Apologies for joining this discussion late. I don't think what you want to do can be done out-of-the-box using the available configurable processors. None of the processors support hierarchical output. There was strong bias toward flattening an event record's data structure in order to make adaption easier when using an output such as Add a Feature to add an event record's data as a new feature in a geodatabase.

We never implemented a workflow which will take a flat data schema and allow you to recombine elements to place them into lists or groups.
Practically speaking, given anything other than the simplest of event record data structures, developing your own custom processor is the best approach.

I mean, you could try to configure a series of out-of-the box Field Calculator processors to take input like:

{ "key_01": "November", "key_02": "Alpha", "key_03": 1714613432 }

And serialize it into a JSON string like:

{ "keys": [{ "key_01": "November" }, { "key_02": "Alpha" }, { "key_03": 1714613432 } ]}

But that's really impractical. The expression needed to do this would look something like:

'{' + ' "keys": ' + ' [{' + ' "key_01": ' + '"' + key_01 + '"' + ' }, ' + '{' + ' "key_02": ' + '"' + key_02 + '"' + ' }, ' + '{' + ' "key_03": ' + key_03 + ' } ' + ']}'

Using simple string concatenation to construct a raw JSON string with more than just a few attribute values is going to become unmanageable very quickly. Especially when you get to the JSON string representation of a polygon with its embedded brackets, commas, and quoted keys. Even if you condense some of what I have above and combine values I've separated as literal strings into fewer single-quoted literal strings ... I was trying to make the illustration somewhat readable ... an out-of-the-box approach just isn't feasible.

Wanted to comment to confirm limitations --
RJ

Re: GeoEvent Field Mapper - from flat to multicardinality

SpatialSean — Thu, 02 May 2024 14:11:58 GMT

I appreciate it - I figured it might have to be a custom processor. Appreciate the help and outlining a potential solution, but as you note it is impractical