Community

compare geoevent field values

4003
6
01-03-2017 07:07 AM
Mikkel_SandermannOlsen
by
New Contributor
‎01-03-2017 07:07 AM

Is it possible to compare a field value in a geoevent with a field value in another geoevent? I wish to make a "correct" datastream that always show the right field values and a "rogue" datastream that have field values that should be compared to the correct one.

As far as I know, tools like incident detector can only compare values between fields in the same geoevent, or a constant value like 30 mph or 100 meters etc.?

Thanks!

0 Kudos
6 Replies
MarkBramer
by
Occasional Contributor II
‎01-03-2017 10:58 AM

mikkeloinformi-dk-esridist

Generally speaking, individual geoevents "stand alone".  GeoEvent Server (new name at 10.5) may process a single geovent, and then not another for seconds or minutes, or months, or longer.  Without introducing state into the mix, these geoevents don't know anything about any other geoevents.  To do what you want, you need to persist geoevent data somewhere, in order for that data to then be compared to other geoevents at a later time.  Generally speaking, you have the following options for persisting geoevent data: an in-memory cache, a file, or a feature service.  I've worked with cache-aware processors and persisting data to feature services in this context. If you want more information, let me know.

Mikkel_SandermannOlsen
by
New Contributor
‎01-03-2017 11:45 PM

Yeah, I’ve tried working with a dummy feature service and use the spatial query extension to append my data, but I’ve not been able to make it work.

If you know of a more obvious/direct solution I would be happy to know more!

My case is this:

I have a maritime radar datastream and a ship AIS datastream.

The radar datastream is considered to always be “correct”, and the AIS data (entered by the skipper) is considered “rogue”.

I wish to be able to monitor these datastreams, and compare them to each other to look for discrepancies (ship size, location, bearing etc.).

To do this I need a method of querying and comparing one datastream to another. As you say, the solution probably lies in a model that has a certain data persistence.

I will be grateful for any knowledge/help you can provide!

/cheers

Fra: Mark Bramer

Sendt: 3. januar 2017 23:26

Til: Mikkel Sandermann Olsen <mikkelo@informi.dk>

Emne: You have been mentioned by Mark Bramer in Re: compare geoevent field values in GeoNet

GeoNet <https://community.esri.com/?et=notification.mention>

You have been mentioned

by Mark Bramer<https://community.esri.com/people/MBramer-esristaff?et=notification.mention> in Re: compare geoevent field values in GeoNet - View Mark Bramer's reference to you<https://community.esri.com/message/656772-re-compare-geoevent-field-values?commentID=656772&et=notification.mention#comment-656772>

0 Kudos
MarkBramer
by
Occasional Contributor II
‎01-05-2017 10:32 AM

mikkeloinformi-dk-esridist‌

Ok, I'm a little familiar with AIS, so let me as some questions:

  • Can the radar feed be thought of as almost a "lookup"?  
  • For example, is it an accurate statement to say that for any message you get from the AIS stream, there's a related record in the radar stream?  
  • If so, would MMSI be the common identifier?  
  • Does each feed have the same schema, or completely different?  
  • If completely different, and there's no MMSI to "join" on, what piece of data is the logical link between the two streams (i.e. "join field")?
  • Is there any way you could send me sample data of each feed?  Event just like a couple records from each, not a large history...

Mark

0 Kudos
Mikkel_SandermannOlsen
by
New Contributor
‎01-09-2017 06:32 AM

  • Can the radar feed be thought of as almost a "lookup"?

  • Yes, it only serves as a comparative verifier of the AIS signals’ validity

  • For example, is it an accurate statement to say that for any message you get from the AIS stream, there's a related record in the radar stream?

  • yes

  • If so, would MMSI be the common identifier?

  • Could be. Right now we are comparing based on geometry

  • Does each feed have the same schema, or completely different?

  • Not exactly similar, but some similarities (vessel size, speed)

  • If completely different, and there's no MMSI to "join" on, what piece of data is the logical link between the two streams (i.e. "join field")?

  • MMSI is probably the way to go on real data but in our tests we run with IMO

  • Is there any way you could send me sample data of each feed? Event just like a couple records from each, not a large history...

  • Sure:

NAME;TONNAGE;LENGTH;DESTINATION;STARTTIME;VESSELTYPE;COUNTRY;CALLSIGN;IMO;LONG;LAT;TIMESTAMP;RADARLONG;RADARLAT;RADARSIZE;RADARID;SENSORTYPE

MARIBEL,42835,22,ANR,03-08-2000 06:00:00,CRUDE OIL TANKER,NO,LAGU6,9326873,16.1900413,54.78525921,03-08-2000 06:00,06:00,16.19004,54.78526,230,MAR,RADAR

MARIBEL,42835,22,ANR,03-08-2000 06:00:00,CRUDE OIL TANKER,NO,LAGU6,9326873,16.1900413,54.78525921,03-08-2000 06:00,06:00,16.19004,54.78526,230,MAR,AIS

MARIBEL,42835,22,ANR,03-08-2000 06:29:00,CRUDE OIL TANKER,NO,LAGU6,9326873,15.64637391,54.74116466,03-08-2000 06:29,06:29,15.64637,54.74116,230,MAR,RADAR

MARIBEL,42835,715,ANR,03-08-2000 06:29:00,CRUDE OIL TANKER,NO,LAGU6,9326873,15.64637391,54.74116466,03-08-2000 06:29,06:29,15.64637,54.74116,230,MAR,AIS

MARIBEL,42835,22,ANR,03-08-2000 06:34:00,CRUDE OIL TANKER,NO,LAGU6,9326873,15.55263815,54.7335573,03-08-2000 06:34,06:34,15.55264,54.73356,230,MAR,RADAR

MARIBEL,42835,22,ANR,03-08-2000 06:34:00,CRUDE OIL TANKER,NO,LAGU6,9326873,15.55263815,54.7335573,03-08-2000 06:34,06:34,15.55264,54.73356,230,MAR,AIS

Currently I am trying to expand on the idea of a dummy radar feature polygonservice, and an AIS geoevent that gets field-enriched by the radardummy and afterwards goes through a comparative analysis.

Fra: Mark Bramer

Sendt: 5. januar 2017 19:33

Til: Mikkel Sandermann Olsen <mikkelo@informi.dk>

Emne: You have been mentioned by Mark Bramer in Re: compare geoevent field values in GeoNet

GeoNet <https://community.esri.com/?et=notification.mention>

You have been mentioned

by Mark Bramer<https://community.esri.com/people/MBramer-esristaff?et=notification.mention> in Re: compare geoevent field values in GeoNet - View Mark Bramer's reference to you<https://community.esri.com/message/657295-re-compare-geoevent-field-values?commentID=657295&et=notification.mention#comment-657295>

0 Kudos
MarkBramer
by
Occasional Contributor II
‎01-10-2017 05:38 AM

Thanks - this makes it clearer.  

Do all records come in pairs, like in the data?  For example I noticed it was RADAR, then AIS, RADAR, AIS, etc.  Not only that, but the data in each pair is nearly identical, which further fed my hunch that the data is arriving in pairs.  If this is true, then it may be easy to use an in-memory processor/filter rather than having to write something to disc or database.

Lastly, is the '715' value for length in the fourth record an example of a 'rogue' value?

0 Kudos
Mikkel_SandermannOlsen
by
New Contributor
‎01-10-2017 06:36 AM

Datarecords neatly stacked in pairs? I wish

Records in the testdata are paired and sequenced after timesteps -great! But in the real application of the system I’m trying to make I need it to work despite a discrepancy in datafeed frequency (eg. radar every minute, AIS every 5 minutes etc) –not so great. This is one of my main headaches, as I just don’t see how to compare two geoevents from different datastreams with varying feed frequencies.

The 715 value is an example of a rogue value, yes.

My current workaround consists of filtering the radardata, making it a polygon geometry and the storing it in a “dummy” service.

Then I use a proximity filter that opens when AIS geoevents intersects with a radardummy polygon (imported via a geofence sync-rule) to enrich the intersecting AIS geoevent with fields from the radardummy.

THEN, I am able to verify my AIS data by comparing the enriched fields from my radardummy with the original AIS fields.. It works.. But it just seems very clunky, and prone to error, and since the sync-rule for my geofences only update every minute, the system will never be as responsive as I would like it to be.

Fra: Mark Bramer

Sendt: 10. januar 2017 14:38

Til: Mikkel Sandermann Olsen <mikkelo@informi.dk>

Emne: Re: - Re: compare geoevent field values

GeoNet <https://community.esri.com/?et=watches.email.thread>

Re: compare geoevent field values

reply from Mark Bramer<https://community.esri.com/people/MBramer-esristaff?et=watches.email.thread> in GeoEvent - View the full discussion<https://community.esri.com/message/658123-re-compare-geoevent-field-values?commentID=658123&et=watches.email.thread#comment-658123>

0 Kudos