GeoEvent: Custom "Flexible" CSV Input Adapter
- Ingesting Variable Length CSV Messages
- Flexible CSV Adapter
- Deploying & Compatibility
- Connectors
- Troubleshooting
Updated to GeoEvent 10.6.1 on June 27, 2022.
Ingesting Variable Length CSV Messages
In GeoEvent, you can use a Text Adapter on an inbound connector to parse comma separated values (CSV), typically with either the TCP transport (like when you use the GeoEvent Simulator) or the File transport (to watch a folder for data files). In general, these CSV messages tend to be static in format allowing you to use one GeoEvent definition to read them in. However, I recently ran into a situation where a device was reporting a truncated message, based on the amount of data it actually had to report. In this case, the devices were only reporting updates, so if there was no update for fields at the end of the message, those fields were omitted from the CSV message. A sample of the data would look like this (I've added some spaces to make the structure a bit clearer):
Name,IdNum,Type,MsgDatetime ,Value1,Value2,Value3,Value4,Value5,Value6,Value7,Value8,Value9,Value10
Red1,1 ,Red ,1585947092133, , , , , ,20.402, , , ,-0.57
Red1,1 ,Red ,1585947692133, , , , , ,1.5001, ,148 ,2
Red1,1 ,Red ,1585948292133, ,10 ,62 ,4 ,-0.22 ,17.701,170 , ,1
Red1,1 ,Red ,1585948892133,6 , ,65 , ,0.64 , ,220 , ,8
Red1,1 ,Red ,1585949492133, , , ,1
Red1,1 ,Red ,1585950092133,94 ,15 , ,1 , , ,480 ,139 ,3
Red1,1 ,Red ,1585950692133,42 ,3 , , ,0.64 , , , ,4 ,-0.99
Red1,1 ,Red ,1585951292133,67 ,-17 , , ,-0.05 ,5.4005, , , ,-0.74
Red1,1 ,Red ,1585951892133, , ,15 ,3 ,-0.72 , ,280 , ,7
Red1,1 ,Red ,1585952492133,22 ,14 , , ,-0.33 ,28.502, , ,8
Red1,1 ,Red ,1585953092133, ,2 , ,1 , , ,395 , ,3
Red1,1 ,Red ,1585953692133, , , ,1 , , , , ,4
Red1,1 ,Red ,1585954292133,55 ,3 ,20 , , , , , ,8
Red1,1 ,Red ,1585954892133,97 ,8 , , ,-0.2 , ,470 , ,8 ,-0.71
Using the OOTB CSV inputs, all but 4 of these messages would be dropped because they don't contain the full set of fields. All of the messages conform to the expected GeoEvent Definition, however some of them are missing the trailing set of fields because there is no updated value for those fields.
Flexible CSV Adapter
To get around this issue I updated the OOTB inbound Text Adapter to be a bit more flexible. For each CSV message, it will attempt to put each field into the provided GeoEvent definition. If there aren't enough fields in the CSV message to fill the GeoEvent, the remaining fields in the event will have their values set to NULL. If there are too many fields in the CSV message, the extra fields in the CSV message will be ignored (like if you only wanted the first 5 values in the above example messages).
Deploying & Compatibility
This adapter is compatible with any version of GeoEvent Server 10.4 or later. To deploy, unzip the attached file and either upload the .jar file to GeoEvent Manager on Site > Components > Adapters using the Add Local Adapter button. Alternatively, you can copy the .jar file into the deploy folder of your GeoEvent Server installation (for windows that would typically be C:\Program Files\ArcGIS\Server\GeoEvent\deploy\ ). The current release is marked version 0.1 (proof of concept). Please report any issues in the comments section of this blog.
Connectors
Adding the .jar file to GeoEvent Server will create two inbound connectors for you. You can create additional connectors utilizing different transports if needed.
Troubleshooting
Loggers that can help troubleshoot:
com.esri.geoevent.adapter.flextext.TextInboundAdapterService
com.esri.geoevent.adapter.flextext.TextInboundAdapter
Source Code
The source code for this adapter is available in the following GitHub repository: