10.4 Changes to the TCP/Text inbound connector -- Construct Geometry From Fields

[RJSunderman]

Changes made to the Text adaptor for the 10.4 product release might catch a few folks off guard, so I want to call attention to what you might expect to see and why.

Several inbound connectors – specifically those using the Generic-JSON and XML adaptors – provide the ability to ‘Construct Geometry From Fields’ so that when event data is received which contains coordinate information on a point location a Geometry can be constructed by the GeoEvent input prior to sending the event to a GeoEvent Service.

At the 10.3.1 release, inputs such as ‘Receive Text from a TCP Socket’ and ‘Watch a Folder for New CSV Files‘ do not have this capability because it was not included in the Text adapter.

It was observed that if a user allowed the input to create a GeoEvent Definition and then later reconfigured the input to request that it construct a Geometry from specified event fields without also editing the event definition to include a Geometry field in which the constructed Geometry could be placed … that the input would stop ingesting event data.

For the 10.4 release, an enhancement was made to the Text adaptor to bring it in line with the Generic-JSON and XML adaptors:

Issue #945:  Text adapter needs to follow Generic-JSON adapter's behavior and create a field of type Geometry, tagged GEOMETRY, when creating a GeoEvent Definition

Now, at the 10.4 release, when you select the ‘Construct Geometry From Fields’ option and specify the name of event attribute fields containing coordinate values, the GeoEvent Definition generated for you by the input will include a field named “Geometry”.  The field will be added as the last field in the event definition and will be tagged with the GEOMETRY GeoTag.

If you leave the ‘Construct Geometry From Fields’ parameter set to its default ‘No’ the generated event definition will not have the Geometry field highlighted above.

If you elect to edit the generated GeoEvent Definition to replace ‘Field1’, ‘Field2’, ‘Field3’ (etc.) with meaningful names, but neglect to include a field in the GeoEvent Definition into which the constructed Geometry can be placed, and then alter your input to switch ‘Construct Geometry From Fields’ from ‘No’ to ‘Yes’ … beginning with the 10.4 release the input will alter the event definition for you to create the needed Geometry field.

Here’s where this enhancement might produce undesired behavior …

Let’s say that you had been using GeoEvent at the 10.3.1 release with a ‘Receive Text from a TCP Socket’ input successfully receiving event data sent from the GeoEvent Simulator. Your GeoEvent Definition might look something like the following:

You export your 10.3.1 GeoEvent product configuration as an XML file, upgrade to 10.4, and import your configuration. When you try simulating the same data which was working at the 10.3.1 release, the input’s event count no longer increments and it does not appear that your GeoEvent is receiving any of the data you’re simulating. You look in the GeoEvent logs and see a message:

Cannot find matching GeoEvent Definition. Event is not created.

The incoming text is SWA2706,3/16/2012 02:25:30 PM,IAD,TPA,B733,37000,-79.585739,34.265521

What’s going on?

It is likely that the field named “Location” is not the field the ‘Construct Geometry From Fields’ operation expects. Even though you were careful to copy and edit the GeoEvent Definition, back in 10.3.1, to include a field whose data type is Geometry and tag the field GEOMETRY … its name is not “Geometry”, so the input doesn’t recognize it.

Remember that delimited text is not well-defined like JSON or XML. If the inbound events contained field names then the adapter would be able to map the received data into the correct fields in the event definition. The 10.4 Text adapter does not assume that a field of type Geometry is sufficient (even the field is tagged GEOMETRY). It looks for a field named “Geometry” and, failing to find one, it logs the error shown above and discards the event.

To fix the problem, all you have to do is edit the GeoEvent Definition imported with your configuration and rename the last field from “Location” to “Geometry”.

This is probably a scenario that many users will encounter when upgrading 10.3.1 configurations which include a TCP/Text input to the 10.4 release. You need to remember that at 10.4 your Geometry fields should be named “Geometry” (not “Location”, or “Target”, or “Geom”). The field must still be the last field in the GeoEvent Definition.

Hope this information saves you a little future grief -

RJ

[RJSunderman]

There’s always more to the story however …

Notice above that I said “you were careful to copy and edit the GeoEvent Definition, back in 10.3.1 …”  Many users do not make a copy of a generated GeoEvent Definition before making edits – they edit the copy technically owned by a GeoEvent component (the input connector in this case). It’s always recommended that you leave a generated event definitions alone. If you need to make edits, copy the event definition and make the edits to a copy you own.

Let’s say that you had edited the generated event definition, exported your entire 10.3.1 product configuration, and then choose ‘Selective Import’ rather than ‘Export Configuration’ to import just your GeoEvent Service from your GeoEvent product configuration XML into your 10.4 installation.

You might notice that your input, output, and GeoEvent Service were imported, but not the GeoEvent Definition you tailored. That is because the input, configured to create a GeoEvent Definition from data it receives, assumes that the previously generated event definition might be stale and it should go ahead and allow the 10.4 adapter to create a new one. Not realizing that the event definition you tailored back in 10.3.1 did not import, you begin simulating data and see event output which does not have a Geometry (notice the trailing comma with no Geometry following the coordinate values):

In this case, the GeoEvent 10.4 input discovered that it needed to generate a GeoEvent Definition, so it did, and it included the needed field named “Geometry”. But the input was configured to look for values in fields named “Longitude” and “Latitude” when creating a Geometry … and the newly generated event definition only has ‘Field1’, ‘Field2’, ‘Field3’ (etc.)

[RJSunderman]

Another potential scenario …

Let’s say that you did make a copy of a GeoEvent Definition in order to rename some … but forgot to add a field named “Geometry” to the event definition you copied and own. You might assume that the input will create the needed “Geometry” field for you … but it doesn’t. You again look in the logs and see error messages to the effect “Cannot find matching GeoEvent Definition. Event is not created.”

When you look back at your input’s configuration you discover that you specified ‘No’, the input should not ‘Create Fixed GeoEvent Definitions’. In this case the input is assuming that it does not have permission to alter a GeoEvent Definition you created and specified the input should use.

The truth is that if you were to specify ‘Yes’ the input should ‘Create Fixed GeoEvent Definitions’, but then key in the name of a GeoEvent Definition you own … the input would assume that it had permission to alter the definition you created. There is a bug open against the 10.4 release that components should not alter GeoEvent Definitions they do not own. The input should be checking to discover that you own the GeoEvent Definition, not the com.esri.ges.adapter.inbound.Text adapter, and conclude that it does not have permission to alter the event definition.

The point is that you should pay particular attention to your input’s configuration and the GeoEvent Definition the input is using. Don’t enter the name of an existing GeoEvent Definition if you configure an input (or other component) to create a GeoEvent Definition for you.

[SharonMeier]

Thank you for posting this.   I am currently dealing with this exact situation.    Quick question - should I delete the GeoEvent definitions that are created by my processors and let them get recreated so that they have the correct "Geometry" name? 

[SharonMeier]

My input connector is reading the input stream and the counter is increasing, as I would expect, but my GeoEvent Service that uses the input connector is not getting the data.   The counter is not increasing. 

[SimonJackson]

RJ Sunderman   I spotted this issue when doing module 2 of the Intro to GeoEvent course (which at the time of writing has not been updated for 10.4).

I tried changing the Flights-TcpTextIn to Geometry as opposed to Location, but still getting the "Cannot find matching GeoEvent Definition. Event is not created. The incoming text is SWA2706,2016-03-07T02:31:25.1933324Z,IAD,TPA,B733,37000,-79.954143,33.654398" in the logs.

What steps need to change for module 2?

[RJSunderman]

Simon -

I'm going to use the 10.3.x-r3 release of the introduction tutorial (the version marked July 2015 and Rev B) with our current 10.4.1 release candidate to try and answer your question about what needs to change in the module 2 narrative and exerices to bring them into line with 10.4.x ...

On page 28, step 8:

• Click to edit the field tagged GEOMETRY and rename the field Location.

You should leave the field named geometry as that is what the 10.4.x release is going to want to see when using the Latitude and Longitude fields to build a point Geomery for you from the Double values received by the TCP/Text input. The field named geometry must still be moved to make it the last field in the GeoEvent Definition.

On page 35, step 3:

• Use the illustration below to configure the properties of the new processor.

You should configure your Field Mapper processor to map the field name you retained (geometry from Flights-TcpTextIn) to the field named geometry (from the Flights event defintion imported from your published feature service's layer).

Your output from the Flights GeoEvent Service should mirror what is illustrated on page 36:

Flights-TcpTextIn,SWA2706,18-Apr-2016 15:21:59,IAD,TPA,B733,37000,-79.585739,34.265521,"-79.585739,34.265521"
Flights,,SWA724,18-Apr-2016 15:29:20,IAD,ABE,SF34,10000,"-76.405289,39.573271"
Flights,,SWA992,18-Apr-2016 15:29:34,IAD,MDW,B737,36400,"-82.067414,39.630957"

The two most recently simulated events are no longer associated with the Flights-TcpTextIn event defintion, they have been mapped to the Flights event definition you imported from your published feature service's layer. The double set of commas correspond to the OBJECTID field left unmapped in your Field Mapper Processor. The Geometry (the last field in the Flights event defintion) holds the point generated for you by the TCP/Text input; its value is represented as a quoted pair of Double values.

Those are the only changes I need to make to the July 2015 and Rev B version of the introduction tutorial to accomodate the 10.4.x behavior I was blogging above.

I have a goal of walking through the entire product introduction tutorial before the UC this summer and updating its narrative and illustrations to reflect the upcoming 10.4.1 product release.

- RJ

[RJSunderman]

Sharon -

Sorry your questions went for so long without a reply. Are the above still issues for you?  Did you perhaps get some help from Esri Tech Support?

You shouldn’t have any problems deleting a GeoEvent Definition created by a processor you’ve incorporated into a GeoEvent Service. Every time I’ve done this the processor has recreated whatever event definition it thinks it needs. Generally I advise folks to copy, rather than edit, a GeoEvent Definition they didn’t create. In this case, because an event definition “upstream” from the processor changed, you could signal the processor to clean-up its managed event definitions by making a simple change to the GeoEvent Service – like removing a connecting line from between two nodes – then publishing the changes to the GeoEvent Service. You can then restore the connecting line you deleted and publish the GeoEvent service again … processors like Field Calculators, GeoTaggers, and Field Enrichers which created event definitions should delete them as part of the first publication and create them once events are received (after the second publication).

If the input’s event count is increasing then I’d assume that event data is being received and successfully adapted to create a GeoEvent. If the GeoEvent Service’s ‘In’ event count is increasing, but not the ‘Out’, I’d assume that events are either being filtered or are dying when a processor node attempts some action and an exception is thrown. On the other hand if neither the ‘In’ event count nor the ‘Out’ event count are increasing I would assume that events are being placed on a queue by the running inbound connector (input) but never being taken off because the RabbitMQ message broker is not running.

- RJ

[SharonMeier]

Hi RJ,

Yes, I did get my issues resolved.