GeoEvent: Consuming New Data
Consuming New Data in GeoEvent
This blog consolidates some notes from other blogs about creating new GeoEvent Definitions when receiving new data. the sections are a bit redundant, but I'll leave it that way since it is important information.
- Consuming New Data in GeoEvent
- JSON/XML Object Name
- Auto-Created GeoEvent Definitions
- Lock Down Your GeoEvent Definitions
- Learning Mode
JSON/XML Object Name
Usually you don't know what kind of data to expect and just need to receive some data before deciding how to parse it. In that case I usually leave the Object Name empty and let GeoEvent try to parse the entire message. Once you know what the data schema looks like, you can come back and modify this property to target the specific section of the incoming data you want to turn into events.
Also note that you may end up identifying one or more sections of the incoming data you want to parse into separate/different events. In that case I would create a second GeoEvent Input for the second section of data I want to parse.
Auto-Created GeoEvent Definitions
I have a habit of making copies of any GeoEvent Definitions that are auto-created for me by GeoEvent modifying the item that auto-created the definition to point to my copy. This allows me to know the exact schema of the data at all times and avoid missing or changing definitions. Because of this, any time I use an item that is automatically going to create the GeoEvent Definition for me, I add a postfix "-auto" to the end of the definition name. This allows me to do two things:
- Identify any GeoEvent Definition that was automatically created (and hence may disappear or change)
- Copy it and rename it without the -auto on the end, so I own that schema and it will be permanent.
Lock Down Your GeoEvent Definitions
Once you receive some data and GeoEvent has helped you determine the schema, you can lock down that schema so it doesn't change and the GeoEvent definition is not transient.
When you have the Create GeoEvent Definition set to Yes, the definition it creates is transient and owned by the input. This means the event can change or even disappear if the input is stopped. Those are things we don't want to happen without us explicitly doing it. To get around this:
- Have the input create the definition for you, but name it something like "MyWebhookName-in-auto".
- After the definition is created by the input, make a copy of it.
- Name the copy something like "MyWebhookName-in" (take the "-auto" off)
- Back in your input, change Create GeoEvent Definition to No and select the new definition you just created.
Learning Mode
Learning mode can be very useful, but dangerous if left on. In some cases where the data you are listening to could be delivered in either partial or different schemas, it might be useful to leave learning mode on for a bit to capture the schema of all the data you might receive. For example,
- If you are receiving events that omit a field when that field is null.
- If you are receiving a mix of different events on a single input feed.
Once you are confident that you've received enough events to complete the schema(s) you are listening for, turn learning mode off. Not turning learning mode off has bit me hard several times so DON'T FORGET TO TURN OFF LEARNING MODE (yelling at myself).
After you have used learning mode to discover your data schema, make one or more copies of the learned schema (see Auto-Created GeoEvent Definitions above). For each copy, delete any unused/needed fields (but make sure you don't change the name!) and save the copy with a descriptive name (like appending "_IN" to the end). Once you've made your copy, make sure you go back to your input and change Create GeoEvent Definition to No and select your copy of the definition.