XML to GeoEvents (information as xml-attributes)

DominikSchlarmann · ‎10-05-2015

@

Dear all,

After working with JSON data, I am now in the situation, where I have to work with XML data using a running instance of ArcGIS Server/GeoEvent Processor 10.3.1.

Find below an XML-data example snippet:

<?xml version="1.0" encoding="utf-8" ?>

<ez_data

xmlns="http://tempuri.org/XML_Echtzeit.xsd" ksType="EPSG:4326">

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez_data>

For tesing purposes I have configured a "Receive XML on a REST Endpoint" input adapter. Later on I will poll an external website for the XML data...

The adapter is configured as the following:

The genereated GeoEvent Defition is generated as the following. The position field is correctly identified as the GEOMETRY-Field, so I assume that I do not need to construct the Geometry-Field manually!?

For tesing-purpose, I've created a GeoEvent Service, which uses the above mentioned input adapter. I use two output-adapters - one is writing the data to a csv file and another one creates a geojson-file.

Coming to my problem: The first and only the first time, the output looks correctly:

The CSV-Output, which is created, looks like the following:

XMLInput,9999,D74692,"{""work"":{""id"":5.0},""timestamp"":1439968807000,""pos"":""{\""x\"":-97.121513,\""y\"":39.798685}""}"

XMLInput,9999,D73512,"{""work"":{""id"":5.0},""timestamp"":1439562583000,""pos"":""{\""x\"":-97.121505,\""y\"":39.79869}""}"

XMLInput,9999,D73514,"{""work"":{""id"":3.0},""timestamp"":1439792663000,""pos"":""{\""x\"":-97.121517,\""y\"":39.798703}""}"

XMLInput,9999,D73679,"{""work"":{""id"":5.0},""timestamp"":1439795974000,""pos"":""{\""x\"":-97.121523,\""y\"":39.798702}""}"

XMLInput,9999,D73517,"{""work"":{""id"":5.0},""timestamp"":1439790820000,""pos"":""{\""x\"":-97.121485,\""y\"":39.798688}""}"

XMLInput,9999,D73515,"{""work"":{""id"":3.0},""timestamp"":1439789327000,""pos"":""{\""x\"":-97.121483,\""y\"":39.798687}""}"

XMLInput,9999,D73510,"{""work"":{""id"":3.0},""timestamp"":1439802384000,""pos"":""{\""x\"":-97.121507,\""y\"":39.798702}""}"

XMLInput,9999,D73516,"{""work"":{""id"":3.0},""timestamp"":1439809374000,""pos"":""{\""x\"":-97.121513,\""y\"":39.798698}""}"

XMLInput,9999,D73508,"{""work"":{""id"":5.0},""timestamp"":1439797832000,""pos"":""{\""x\"":-97.121507,\""y\"":39.798705}""}"

Also the geojson-file is created including the geometry field, so I would be able to visualize the data on geojson.io.

Check the output here: [JSON] geojson-output - Pastebin.com

Now, when I push the the above mentioned XML-data again to my RestInput-Adapter, the geometry-field is not anymore recognized, so the output look like this:

XMLInput,9999,A54692,"{""work"":{""id"":5.0},""timestamp"":1439968807000}"

XMLInput,9999,A53515,"{""work"":{""id"":3.0},""timestamp"":1439789327000}"

XMLInput,9999,A53679,"{""work"":{""id"":5.0},""timestamp"":1439795974000}"

XMLInput,9999,A53512,"{""work"":{""id"":5.0},""timestamp"":1439562583000}"

XMLInput,9999,A53516,"{""work"":{""id"":3.0},""timestamp"":1439809374000}"

XMLInput,9999,A53510,"{""work"":{""id"":3.0},""timestamp"":1439802384000}"

XMLInput,9999,A53514,"{""work"":{""id"":3.0},""timestamp"":1439792663000}"

XMLInput,9999,A53517,"{""work"":{""id"":5.0},""timestamp"":1439790820000}"

XMLInput,9999,A53508,"{""work"":{""id"":5.0},""timestamp"":1439797832000}"

Also the geojson-output is not created correctly as the geometry-object is not generated anymore.

Coming to my questions:

Why my Workflow is only working for the very first time?

Would be awesome to get some feedback!

All the best,

Dominik

DominikSchlarmann · ‎10-19-2015

I've finally solved the problem by myself. Find below a short description of those steps, which I've performed. Maybe this is useful for people with a similar problem:

As you can see in the example data (check the first post), the geometry information (x and y) are stored as XML attributes and not as a separate XML-elements. I think, dealing with XML-elements is a little bit easier and more straight forward as the GeoEvent Processor easily creates a correct GeoEvent definition, where each field represents a separate XML-element.

Nevertheless...using the XML-example-snippet from the first post, the GeoEventProcessor would create a similar GeoEvent Definition, but for any reason (maybe RJ Sunderman has some additional information) the GeoEvent Processor does not construct the right GeoEvent Definition, which is needed for using the geometry values (as you can see in the first screenshot of my first post, the x and y field are missing)

These are the steps, which I've to perform:

Configure your Input adapter in the way (for my use case I have used an "Receive XML on a REST Endpoint"), that a GeoEvent Definition is created automatically.
Go to "Site -> GeoEvent Definition", search the GeoEvent Definition, which was created by your Input-Adapter, and start the editing-form.
1. For those elements, which are missing (in my case the x and y coordinate were not processed), add the necessary fields. Ensure that the structure is still valid. For my use case, I need to edit the pos-object: I've changed the type to "group" and added a x and y field as doubles to the previous changed pos-group.
2. Add another field to the "root" of your GeoEvent Definition (for example "geometryFromCoordinates") and tag this with the "Geometry-Tag".
3. Save your changes.

Switch back to your Input-Adapter and change the following
1. For the Setting "Create GeoEvent Definition" switch from "Yes" to "No" and select the GeoEvent Definition, which you have edited before.
2. For the Setting "Construct Geometry From Fields" switch from "No" to "Yes" and specify the fields for the geometry. In my case the x-field is defined by ez.pos.x and the y-field by ez.pos.y.
3. Save the changes.
You're done - or at least I was done and everything worked and works like a charm.
(optional): For easier processing, I have created another GeoEvent Definition without the nested structure of ez-element. In combination with a Field Mapper it is now straight forward to process the data.

All the best,

Dominik

View solution in original post

DominikSchlarmann · ‎10-15-2015

Has anyone a hint? Maybe RJ Sunderman ? I've already tried things like the "Field Calculator (regular expressions)", but unfortunately I am still not able to use the above outlined xml as an input; of course it might be caused by an incorrect usage of the Field Calculator (regular expression) processor.

DominikSchlarmann · ‎10-19-2015

I've finally solved the problem by myself. Find below a short description of those steps, which I've performed. Maybe this is useful for people with a similar problem:

As you can see in the example data (check the first post), the geometry information (x and y) are stored as XML attributes and not as a separate XML-elements. I think, dealing with XML-elements is a little bit easier and more straight forward as the GeoEvent Processor easily creates a correct GeoEvent definition, where each field represents a separate XML-element.

Nevertheless...using the XML-example-snippet from the first post, the GeoEventProcessor would create a similar GeoEvent Definition, but for any reason (maybe RJ Sunderman has some additional information) the GeoEvent Processor does not construct the right GeoEvent Definition, which is needed for using the geometry values (as you can see in the first screenshot of my first post, the x and y field are missing)

These are the steps, which I've to perform:

Configure your Input adapter in the way (for my use case I have used an "Receive XML on a REST Endpoint"), that a GeoEvent Definition is created automatically.
Go to "Site -> GeoEvent Definition", search the GeoEvent Definition, which was created by your Input-Adapter, and start the editing-form.
1. For those elements, which are missing (in my case the x and y coordinate were not processed), add the necessary fields. Ensure that the structure is still valid. For my use case, I need to edit the pos-object: I've changed the type to "group" and added a x and y field as doubles to the previous changed pos-group.
2. Add another field to the "root" of your GeoEvent Definition (for example "geometryFromCoordinates") and tag this with the "Geometry-Tag".
3. Save your changes.

Switch back to your Input-Adapter and change the following
1. For the Setting "Create GeoEvent Definition" switch from "Yes" to "No" and select the GeoEvent Definition, which you have edited before.
2. For the Setting "Construct Geometry From Fields" switch from "No" to "Yes" and specify the fields for the geometry. In my case the x-field is defined by ez.pos.x and the y-field by ez.pos.y.
3. Save the changes.
You're done - or at least I was done and everything worked and works like a charm.
(optional): For easier processing, I have created another GeoEvent Definition without the nested structure of ez-element. In combination with a Field Mapper it is now straight forward to process the data.

All the best,

Dominik

GregCarr-Hyde · ‎10-19-2015

Hi Dominik

Thanks for posting the answer to this I've been looking everywhere for how to handle an XML input and your screenshots have cleared up a lot of my questions.

All the best

Greg

DominikSchlarmann · ‎10-19-2015

Hi Greg,

You're welcome! I am happy that my information are helpful for you

If you have any additional questions: Feel free to ask. Maybe I can help....

Best,

Dominik

GregCarr-Hyde · ‎10-19-2015

Thanks Dominik,

I do have a question that maybe you can help with. Do you know a URL that I can use as test for polling XML.

I'm looking at NOAA at the moment NuSOAP: ndfdXML

But not really sure what to set as the main URL for their information.

Thanks

Greg

DominikSchlarmann · ‎10-19-2015

Dear Greg Carr-Hyde,

I'm not 100% sure if I have understood you correctly: Do you search for a way to expose the data provided by NOAA in XML-Fomat?

If yes, find my answer in the following lines:

As far as I see, the url you've provided in your post (NuSOAP: ndfdXML) is not the right one. The URL describes the SOAP-interface and the usage of this. This means, you can and have to use this interface using the soap-protocol.

From my point of view, SOAP is not an usable option for the GeoEvent Processor; at least not with the provided adapters. Possibly you could implement an adapter by your own, which could be realized using the the GeoEvent SDK.

Another option, which is a lot of easier, might be the following:

I've looked at the NOAA website and they also offer a REST-interface for their data: National Digital Forecast Database XML/REST Service - NOAA's National Weather Service

There are also some examples (which you have to edit as most of them are using a time-interval, which is too old) and even a "query builder".

A working example is the following: http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdXMLclient.php?lat1=35.00&lon1...

Using those links, it should be no problem to their data with the GeoEvent Processor.

Hopefully the information a somehow helpful for you.

All the best,

Dominik

GregCarr-Hyde · ‎10-19-2015

Hi Dominik

Thank you very much for your answer. You have understood me perfectly

I have to implement a truck tracking process but the company that provides the data does so via a SOAP interface (I have asked if an XML rest end point is possible and the answer is no). So I've been trying to simulate the scenario with the NOAA data. But as you say SOAP is not ideal.

I will need to ask our developers to assist with creating a custom processor with the GeoEvntSDK but I was hoping there was a more simple solution.

Thank you so much for the guidance.

Kind regards

Greg

RJSunderman · ‎01-21-2016

Greg Carr-Hyde - Here are a couple of links to other discussion threads which provide some background on SOAP. The SOAP protocol does use XML to structure its requests and responses, but SOAP messages are not the type of event structure GeoEvent is expecting when XML is used to format event data provided by a real-time data stream.

The sample XML Dominik Schlarmann provides at the top of this thread is a much better example of an XML event structure. GeoEvent can work to translate this XML into a JSON format (an implementation detail handled for you by the XML adapter) in order to ingest the event data stream. I mention this because JSON does not support the concepts of elements and attributes that XML supports This can present you with challenges when bringing in complex XML data structures. Generally speaking, if you must use XML, I would avoid XML which includes attributes. Stick to using elements and the XML-to-JSON translation will work better.

Sorry this response is coming so late.

Hope the information is helpful.

- RJ

See Also: W3 Schools - XML Elements vs. Attributes

RJSunderman · ‎01-21-2016

Dominik Schlarmann - Glad to see you were able to identify the issue. My apologies for the frustration and time you had to spend to figure it out. Thanks for posting your workflow steps. They are spot on for what you need to do in order to bring the 'x' and 'y' attribute values in as Double values and have the GeoEvent input use the coordinate values to construct a true Geometry.

It is not at all obvious, but a Geometry is not a simple Group structure, as you discovered. Sometimes when GeoEvent examines a data structure in order to makes its best guess at how a GeoEvent Definition ought to be created, a group with coordinate values which looks enough like a Geometry ends up being represented as a Geometry in the generated event definition, forcing you to go back and correct the generated event definition to handle the nested coordinate values as sub-elements within a group.

Here's a couple of examples of what different Geometries look like in a JSON format. Notice that the polygon's coordinates are organized as nested lists with a keyword "rings".

The above is why the GeoEvent inputs which allow you to construct Geometry from attributes only support the creation of Point Geometry. You can find additional detail on working with Geometries in GeoEvent in 'Appendix A' of the GeoEvent Extension Introduction tutorial. Look for the section titled "Using a Field Calculator Processor to compute a geometry".