topic Re: XML to GeoEvents (information as xml-attributes) in ArcGIS GeoEvent Server Questions

XML to GeoEvents (information as xml-attributes)

DominikSchlarmann — Mon, 05 Oct 2015 13:59:18 GMT

Dear all,

After working with JSON data, I am now in the situation, where I have to work with XML data using a running instance of ArcGIS Server/GeoEvent Processor 10.3.1.

Find below an XML-data example snippet:

<?xml version="1.0" encoding="utf-8" ?>

<ez_data

xmlns="http://tempuri.org/XML_Echtzeit.xsd" ksType="EPSG:4326">

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez_data>

For tesing purposes I have configured a "Receive XML on a REST Endpoint" input adapter. Later on I will poll an external website for the XML data...

The adapter is configured as the following:

The genereated GeoEvent Defition is generated as the following. The position field is correctly identified as the GEOMETRY-Field, so I assume that I do not need to construct the Geometry-Field manually!?

For tesing-purpose, I've created a GeoEvent Service, which uses the above mentioned input adapter. I use two output-adapters - one is writing the data to a csv file and another one creates a geojson-file.

Coming to my problem: The first and only the first time, the output looks correctly:

The CSV-Output, which is created, looks like the following:

XMLInput,9999,D74692,"{""work"":{""id"":5.0},""timestamp"":1439968807000,""pos"":""{\""x\"":-97.121513,\""y\"":39.798685}""}"

XMLInput,9999,D73512,"{""work"":{""id"":5.0},""timestamp"":1439562583000,""pos"":""{\""x\"":-97.121505,\""y\"":39.79869}""}"

XMLInput,9999,D73514,"{""work"":{""id"":3.0},""timestamp"":1439792663000,""pos"":""{\""x\"":-97.121517,\""y\"":39.798703}""}"

XMLInput,9999,D73679,"{""work"":{""id"":5.0},""timestamp"":1439795974000,""pos"":""{\""x\"":-97.121523,\""y\"":39.798702}""}"

XMLInput,9999,D73517,"{""work"":{""id"":5.0},""timestamp"":1439790820000,""pos"":""{\""x\"":-97.121485,\""y\"":39.798688}""}"

XMLInput,9999,D73515,"{""work"":{""id"":3.0},""timestamp"":1439789327000,""pos"":""{\""x\"":-97.121483,\""y\"":39.798687}""}"

XMLInput,9999,D73510,"{""work"":{""id"":3.0},""timestamp"":1439802384000,""pos"":""{\""x\"":-97.121507,\""y\"":39.798702}""}"

XMLInput,9999,D73516,"{""work"":{""id"":3.0},""timestamp"":1439809374000,""pos"":""{\""x\"":-97.121513,\""y\"":39.798698}""}"

XMLInput,9999,D73508,"{""work"":{""id"":5.0},""timestamp"":1439797832000,""pos"":""{\""x\"":-97.121507,\""y\"":39.798705}""}"

Also the geojson-file is created including the geometry field, so I would be able to visualize the data on geojson.io.

Check the output here: [JSON] geojson-output - Pastebin.com

Now, when I push the the above mentioned XML-data again to my RestInput-Adapter, the geometry-field is not anymore recognized, so the output look like this:

XMLInput,9999,A54692,"{""work"":{""id"":5.0},""timestamp"":1439968807000}"

XMLInput,9999,A53515,"{""work"":{""id"":3.0},""timestamp"":1439789327000}"

XMLInput,9999,A53679,"{""work"":{""id"":5.0},""timestamp"":1439795974000}"

XMLInput,9999,A53512,"{""work"":{""id"":5.0},""timestamp"":1439562583000}"

XMLInput,9999,A53516,"{""work"":{""id"":3.0},""timestamp"":1439809374000}"

XMLInput,9999,A53510,"{""work"":{""id"":3.0},""timestamp"":1439802384000}"

XMLInput,9999,A53514,"{""work"":{""id"":3.0},""timestamp"":1439792663000}"

XMLInput,9999,A53517,"{""work"":{""id"":5.0},""timestamp"":1439790820000}"

XMLInput,9999,A53508,"{""work"":{""id"":5.0},""timestamp"":1439797832000}"

Also the geojson-output is not created correctly as the geometry-object is not generated anymore.

Coming to my questions:

Why my Workflow is only working for the very first time?

Would be awesome to get some feedback!

All the best,

Dominik

Re: XML to GeoEvents (XML handling)

DominikSchlarmann — Thu, 15 Oct 2015 12:55:48 GMT

Has anyone a hint? Maybe RJ Sunderman ? I've already tried things like the "Field Calculator (regular expressions)", but unfortunately I am still not able to use the above outlined xml as an input; of course it might be caused by an incorrect usage of the Field Calculator (regular expression) processor.

Re: XML to GeoEvents (information as xml-attributes)

DominikSchlarmann — Mon, 19 Oct 2015 13:36:22 GMT

I've finally solved the problem by myself. Find below a short description of those steps, which I've performed. Maybe this is useful for people with a similar problem:

As you can see in the example data (check the first post), the geometry information (x and y) are stored as XML attributes and not as a separate XML-elements. I think, dealing with XML-elements is a little bit easier and more straight forward as the GeoEvent Processor easily creates a correct GeoEvent definition, where each field represents a separate XML-element.

Nevertheless...using the XML-example-snippet from the first post, the GeoEventProcessor would create a similar GeoEvent Definition, but for any reason (maybe RJ Sunderman has some additional information) the GeoEvent Processor does not construct the right GeoEvent Definition, which is needed for using the geometry values (as you can see in the first screenshot of my first post, the x and y field are missing)

These are the steps, which I've to perform:

Configure your Input adapter in the way (for my use case I have used an "Receive XML on a REST Endpoint"), that a GeoEvent Definition is created automatically.
Go to "Site -> GeoEvent Definition", search the GeoEvent Definition, which was created by your Input-Adapter, and start the editing-form.
1. For those elements, which are missing (in my case the x and y coordinate were not processed), add the necessary fields. Ensure that the structure is still valid. For my use case, I need to edit the pos-object: I've changed the type to "group" and added a x and y field as doubles to the previous changed pos-group.
2. Add another field to the "root" of your GeoEvent Definition (for example "geometryFromCoordinates") and tag this with the "Geometry-Tag".
3. Save your changes.

Switch back to your Input-Adapter and change the following
1. For the Setting "Create GeoEvent Definition" switch from "Yes" to "No" and select the GeoEvent Definition, which you have edited before.
2. For the Setting "Construct Geometry From Fields" switch from "No" to "Yes" and specify the fields for the geometry. In my case the x-field is defined by ez.pos.x and the y-field by ez.pos.y.
3. Save the changes.
You're done - or at least I was done and everything worked and works like a charm.
(optional): For easier processing, I have created another GeoEvent Definition without the nested structure of ez-element. In combination with a Field Mapper it is now straight forward to process the data.

All the best,

Dominik

Re: XML to GeoEvents (information as xml-attributes)

GregCarr-Hyde — Mon, 19 Oct 2015 13:45:28 GMT

Hi Dominik

Thanks for posting the answer to this I've been looking everywhere for how to handle an XML input and your screenshots have cleared up a lot of my questions.

All the best

Greg

Re: XML to GeoEvents (information as xml-attributes)

DominikSchlarmann — Mon, 19 Oct 2015 13:54:56 GMT

Hi Greg,

You're welcome! I am happy that my information are helpful for you

If you have any additional questions: Feel free to ask. Maybe I can help....

Best,

Dominik

Re: XML to GeoEvents (information as xml-attributes)

GregCarr-Hyde — Mon, 19 Oct 2015 14:00:01 GMT

Thanks Dominik,

I do have a question that maybe you can help with. Do you know a URL that I can use as test for polling XML.

I'm looking at NOAA at the moment NuSOAP: ndfdXML

But not really sure what to set as the main URL for their information.

Thanks

Greg

Re: XML to GeoEvents (information as xml-attributes)

DominikSchlarmann — Mon, 19 Oct 2015 18:39:58 GMT

Dear Greg Carr-Hyde,

I'm not 100% sure if I have understood you correctly: Do you search for a way to expose the data provided by NOAA in XML-Fomat?

If yes, find my answer in the following lines:

As far as I see, the url you've provided in your post (NuSOAP: ndfdXML) is not the right one. The URL describes the SOAP-interface and the usage of this. This means, you can and have to use this interface using the soap-protocol.

From my point of view, SOAP is not an usable option for the GeoEvent Processor; at least not with the provided adapters. Possibly you could implement an adapter by your own, which could be realized using the the GeoEvent SDK.

Another option, which is a lot of easier, might be the following:

I've looked at the NOAA website and they also offer a REST-interface for their data: National Digital Forecast Database XML/REST Service - NOAA's National Weather Service

There are also some examples (which you have to edit as most of them are using a time-interval, which is too old) and even a "query builder".

A working example is the following: http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdXMLclient.php?lat1=35.00&lon1=-82.00&lat2=35.5&lon2=-81.50&resolutionSub=20.0&product=time-series&begin=20015-10-20T00:00:00&end=2015-10-21T00:00:00&maxt=maxt&mint=mint

Using those links, it should be no problem to their data with the GeoEvent Processor.

Hopefully the information a somehow helpful for you.

All the best,

Dominik

Re: XML to GeoEvents (information as xml-attributes)

GregCarr-Hyde — Tue, 20 Oct 2015 06:11:18 GMT

Hi Dominik

Thank you very much for your answer. You have understood me perfectly

I have to implement a truck tracking process but the company that provides the data does so via a SOAP interface (I have asked if an XML rest end point is possible and the answer is no). So I've been trying to simulate the scenario with the NOAA data. But as you say SOAP is not ideal.

I will need to ask our developers to assist with creating a custom processor with the GeoEvntSDK but I was hoping there was a more simple solution.

Thank you so much for the guidance.

Kind regards

Greg

Re: XML to GeoEvents (information as xml-attributes)

MoizIshfaq — Thu, 24 Dec 2015 16:47:10 GMT

Dear Dominik,

First of all thanks for the valuable post as documentation on 'GeoEvent Poll XML form external website' is limited. I was able to pull information from a couple of xml sources but am struggling with extracting information from a multi-level hierarchical xml structure as attached to this message.

My first question is if I can extract all the location information defined at the lowest hierarchical position along with high level details from the top level xml hierarchical nodes and feed it into a standard feature service. My current experiment either extract one record if I select the top level node as 'XML Object Name' or 8 records if I select the 5th level node i.e., 'Location', but lose other high level information in the later scenario. Can you kindly provide any insight about it?

My second question is about having more than one objects inside an 'Issue' element but GeoEvent is only picking a single one from it, though I selected 'Location' as my 'XML Object Name'. Can we somehow extract all the location objects with all the elements and attribute information from this xml?

In advance I thank you for your support and time.

Re: XML to GeoEvents (information as xml-attributes)

RJSunderman — Thu, 21 Jan 2016 20:11:30 GMT

Dominik Schlarmann - Glad to see you were able to identify the issue. My apologies for the frustration and time you had to spend to figure it out. Thanks for posting your workflow steps. They are spot on for what you need to do in order to bring the 'x' and 'y' attribute values in as Double values and have the GeoEvent input use the coordinate values to construct a true Geometry.

It is not at all obvious, but a Geometry is not a simple Group structure, as you discovered. Sometimes when GeoEvent examines a data structure in order to makes its best guess at how a GeoEvent Definition ought to be created, a group with coordinate values which looks enough like a Geometry ends up being represented as a Geometry in the generated event definition, forcing you to go back and correct the generated event definition to handle the nested coordinate values as sub-elements within a group.

Here's a couple of examples of what different Geometries look like in a JSON format. Notice that the polygon's coordinates are organized as nested lists with a keyword "rings".

The above is why the GeoEvent inputs which allow you to construct Geometry from attributes only support the creation of Point Geometry. You can find additional detail on working with Geometries in GeoEvent in 'Appendix A' of the GeoEvent Extension Introduction tutorial. Look for the section titled "Using a Field Calculator Processor to compute a geometry".

Re: XML to GeoEvents (information as xml-attributes)

RJSunderman — Thu, 21 Jan 2016 20:34:58 GMT

Greg Carr-Hyde - Here are a couple of links to other discussion threads which provide some background on SOAP. The SOAP protocol does use XML to structure its requests and responses, but SOAP messages are not the type of event structure GeoEvent is expecting when XML is used to format event data provided by a real-time data stream.

The sample XML Dominik Schlarmann provides at the top of this thread is a much better example of an XML event structure. GeoEvent can work to translate this XML into a JSON format (an implementation detail handled for you by the XML adapter) in order to ingest the event data stream. I mention this because JSON does not support the concepts of elements and attributes that XML supports This can present you with challenges when bringing in complex XML data structures. Generally speaking, if you must use XML, I would avoid XML which includes attributes. Stick to using elements and the XML-to-JSON translation will work better.

Sorry this response is coming so late.

Hope the information is helpful.

- RJ

Re: XML to GeoEvents (information as xml-attributes)

RJSunderman — Thu, 21 Jan 2016 21:28:02 GMT

Hello Britt -

I downloaded the sample file.xml you attached. I had to remove some literal dash characters ('-') from several lines before an XML editor I normally use would validate and display the file's content for me to review.

My first question is if I can extract all the location information defined at the lowest hierarchical position along with high level details from the top level xml hierarchical nodes ...

Let me make sure I understand what you want to do. Given a structure like the one illustrated below, you want to bring each <Issue> element in from the <Issues> list as separate events, but you want the events to incorporate the information from the <LocalTime> and/or the <UtcTime> elements found outside the <Issues> structure.

The quick answer is no, you cannot do this. When specifying the 'XML Object Name' parameter for either the 'Poll an External Website for XML' or 'Receive XML on a REST Endpoint' GeoEvent inputs, you are telling GeoEvent that it should scan forward (through the XML structure), skipping past elements and attributes until it locates the specified 'XML Object Name' node. GeoEvent will then bring in elements listed beneath that root node as separate events. The trade-off to identifying a list deep within a structure and having GeoEvent parse that list is that you have told GeoEvent to skip past and disregard elements outside of the list.

You might want to take a look at the discussion thread XML can I use this source? The problem presented in that thread is similar - the date on which each event occurred is outside of the list of events Since each record in the list only specifies a time, you need a way to enrich each of the individual events with the date recorded as an attribute in the list's root node. In your case your challenge is compounded as it looks like each <Issue> can have more than one <Location>.

... I selected 'Location' as my 'XML Object Name'. Can we somehow extract all the location objects with all the elements and attribute information from this xml?

Looking at the XML structure, <Issues> appears to be a list. I would expect I could use GeoEvent to parse the <Issues> structure in order to bring each <Issue> in as a discrete event. Because the various <Location> records are not organized beneath a single root node (e.g. thay are children associated with a particular issue) I don't think the data structure will support GeoEvent bringing each <Location> in as a separate event.

The idea behind the 'XML Object Name' parameter is that you can specify a single node as a "root" node and have GeoEvent only bring in data found beneath that "root" node as discrete (separate) events.

Hope this information is helpful -