Problem with 'Watch a folder for new .csv files'

AndreasEspersen

Hi all,

I have a problem with getting data from a CSV file processed by GEP that is beginning to drive me mad.....!

I'm using the input Watch a folder for new .csv files but I haven't had any luck getting the GEP to output it correctly. Either it is only showing the first line or else it shows nothing when outputting to Publish text to a TCP Socket and gives various errors in the log that haven't guided me in the right direction.

The input to GEP is made via model builder and runs every 5 minutes and an example of the file can be seen attached (The file format was changed to txt so the file could be uploaded.)

My Geoevent Definitions and input adaptor can be seen in the attached PNGs.

When changing the GeoEvent Definition to 'incident', which comes with the installation and using the csv files (Flights/Trucks) from the tutorial things are working.

I think the problem is related to building the geometry but i havent found a solution for it.

Any ideas on what is going wrong would be greatly appreciated.

Regards

Andy

InputAdaptor.PNG

GeoEventDefinition.PNG

input.txt.zip

DougBrowning_0-1753368282039.png

L77_0-1753366093342.png

Brian_McLeer_0-1753362131794.png

Anonymous User

Original User: RJ Sunderman

Hello Andreas -

I've a few thoughts on what you might be experiencing.

First, I would remove the line specifying the attribute field names from the CSV file you are writing into the folder the 'Watch a folder for new .csv files' inbound connector is watching. Given a specified GeoEvent Definition, I observed the 10.2.2 product placing <null> values into GeoEvent fields when the inbound connector's adapter could not successfully parse a string value from the system text file to create the specified type. The field name "XCoord" for example cannot be converted to a value of type 'Double'. It might be that GeoEvent Processor can handle the header row - but it would still be better to include only the actual event data in the file and leave out the attribute field names. Also, I would be very careful to make sure that the event attributes being received as delimited values are received in the order specified by the GeoEvent Definition. You have at least a couple dozen attribute fields in your input, and it's easy to introduce an off-by-one error that will result in the received input not matching the event definition and the adapter not being able to parse the data.

Second, and this has caught me before, make sure that your system file contains an empty line at the end of the input file. Since you only have one line of actual data, if there is not an actual newline (\n) terminating the line of delimited data values, the GeoEvent Processor's adapter will buffer input every time a file is read until it finally finds a newline. A good way to avoid this is to always use input files with at least a couple of lines of data ... then you only end up wondering why the *last* line in the file isn't being processed.

Third, it took me a couple of minutes to recognize that the numeric values in your sample text file were expressed using the international convention with a ',' rather than a '.' in the decimal values. Your GeoEvent Definition specifies that the first six data values should be received as Double, Double, Integer, Double, Short, Date:

[ATTACH=CONFIG]35053[/ATTACH]
Click to Enlarge

When told to expect a Double value, the Text adapter out-of-the-box actually replaces ',' with '.' and brings in the value 11,19770167 as a numeric double: 11.19770167 ... I was a bit surprised by this actually.

Looking at the first six values from your sample, I can construct a GeoEvent Definition which successfully brings in these values.

11,19770167;56,58325000;219016873;20,000000;1;2014-07-01 10:44:44

Finally, if you want the the out-of-the-box Text adapter to create a Geometry from coordinate values received with the event ... unless you specify a WKID (Well Known Identifier) for the projected coordinate system associated with the coordinate values ... the coordinates must be expressed in decimal degrees.

Since you're GeoEvent Defintion is bringing the values in as Double, I'm going to assume that the value 11.19770167 is 'Latitude' and the value 56.58325000 is 'Longitude' ... so you don't need to specify a WKID, but you do need an "extra" field in your GeoEvent Definition of type Geometry into which the Text adapter can place the Geometry is creates. So your event definition will have one (1) more field than the number of attributes in your delimited list.

Keeping with the simple input example in the code block above, if I add an event field 'Location' to my GeoEvent Definition, I tried to have the input create a Geometry for me using the first two event attributes as the coordinate values ... and this is where I ran into trouble.

[ATTACH=CONFIG]35054[/ATTACH]
Click to Enlarge

[ATTACH=CONFIG]35055[/ATTACH]
Click to Enlarge

Evidently the adapter, told to expect a Double, is willing to take a value 11,19770167 and convert it to a numeric value 11.19770167 ... but it is unwilling to take the same value and use it to create a Geometry. I will follow-up with an example of how you can use a Field Calculator processor to work around this limitation.

Hope this information helps -
RJ

Capture.PNG

Capture2.PNG

Capture3.PNG