XML to GeoEvents (information as xml-attributes)

DominikSchlarmann · ‎10-05-2015

@

Dear all,

After working with JSON data, I am now in the situation, where I have to work with XML data using a running instance of ArcGIS Server/GeoEvent Processor 10.3.1.

Find below an XML-data example snippet:

<?xml version="1.0" encoding="utf-8" ?>

<ez_data

xmlns="http://tempuri.org/XML_Echtzeit.xsd" ksType="EPSG:4326">

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez>

</fz>

</ez_data>

For tesing purposes I have configured a "Receive XML on a REST Endpoint" input adapter. Later on I will poll an external website for the XML data...

The adapter is configured as the following:

The genereated GeoEvent Defition is generated as the following. The position field is correctly identified as the GEOMETRY-Field, so I assume that I do not need to construct the Geometry-Field manually!?

For tesing-purpose, I've created a GeoEvent Service, which uses the above mentioned input adapter. I use two output-adapters - one is writing the data to a csv file and another one creates a geojson-file.

Coming to my problem: The first and only the first time, the output looks correctly:

The CSV-Output, which is created, looks like the following:

XMLInput,9999,D74692,"{""work"":{""id"":5.0},""timestamp"":1439968807000,""pos"":""{\""x\"":-97.121513,\""y\"":39.798685}""}"

XMLInput,9999,D73512,"{""work"":{""id"":5.0},""timestamp"":1439562583000,""pos"":""{\""x\"":-97.121505,\""y\"":39.79869}""}"

XMLInput,9999,D73514,"{""work"":{""id"":3.0},""timestamp"":1439792663000,""pos"":""{\""x\"":-97.121517,\""y\"":39.798703}""}"

XMLInput,9999,D73679,"{""work"":{""id"":5.0},""timestamp"":1439795974000,""pos"":""{\""x\"":-97.121523,\""y\"":39.798702}""}"

XMLInput,9999,D73517,"{""work"":{""id"":5.0},""timestamp"":1439790820000,""pos"":""{\""x\"":-97.121485,\""y\"":39.798688}""}"

XMLInput,9999,D73515,"{""work"":{""id"":3.0},""timestamp"":1439789327000,""pos"":""{\""x\"":-97.121483,\""y\"":39.798687}""}"

XMLInput,9999,D73510,"{""work"":{""id"":3.0},""timestamp"":1439802384000,""pos"":""{\""x\"":-97.121507,\""y\"":39.798702}""}"

XMLInput,9999,D73516,"{""work"":{""id"":3.0},""timestamp"":1439809374000,""pos"":""{\""x\"":-97.121513,\""y\"":39.798698}""}"

XMLInput,9999,D73508,"{""work"":{""id"":5.0},""timestamp"":1439797832000,""pos"":""{\""x\"":-97.121507,\""y\"":39.798705}""}"

Also the geojson-file is created including the geometry field, so I would be able to visualize the data on geojson.io.

Check the output here: [JSON] geojson-output - Pastebin.com

Now, when I push the the above mentioned XML-data again to my RestInput-Adapter, the geometry-field is not anymore recognized, so the output look like this:

XMLInput,9999,A54692,"{""work"":{""id"":5.0},""timestamp"":1439968807000}"

XMLInput,9999,A53515,"{""work"":{""id"":3.0},""timestamp"":1439789327000}"

XMLInput,9999,A53679,"{""work"":{""id"":5.0},""timestamp"":1439795974000}"

XMLInput,9999,A53512,"{""work"":{""id"":5.0},""timestamp"":1439562583000}"

XMLInput,9999,A53516,"{""work"":{""id"":3.0},""timestamp"":1439809374000}"

XMLInput,9999,A53510,"{""work"":{""id"":3.0},""timestamp"":1439802384000}"

XMLInput,9999,A53514,"{""work"":{""id"":3.0},""timestamp"":1439792663000}"

XMLInput,9999,A53517,"{""work"":{""id"":5.0},""timestamp"":1439790820000}"

XMLInput,9999,A53508,"{""work"":{""id"":5.0},""timestamp"":1439797832000}"

Also the geojson-output is not created correctly as the geometry-object is not generated anymore.

Coming to my questions:

Why my Workflow is only working for the very first time?

Would be awesome to get some feedback!

All the best,

Dominik

MoizIshfaq · ‎12-24-2015

Dear Dominik,

First of all thanks for the valuable post as documentation on 'GeoEvent Poll XML form external website' is limited. I was able to pull information from a couple of xml sources but am struggling with extracting information from a multi-level hierarchical xml structure as attached to this message.

My first question is if I can extract all the location information defined at the lowest hierarchical position along with high level details from the top level xml hierarchical nodes and feed it into a standard feature service. My current experiment either extract one record if I select the top level node as 'XML Object Name' or 8 records if I select the 5th level node i.e., 'Location', but lose other high level information in the later scenario. Can you kindly provide any insight about it?

My second question is about having more than one objects inside an 'Issue' element but GeoEvent is only picking a single one from it, though I selected 'Location' as my 'XML Object Name'. Can we somehow extract all the location objects with all the elements and attribute information from this xml?

In advance I thank you for your support and time.

RJSunderman · ‎01-21-2016

Hello Britt -

I downloaded the sample file.xml you attached. I had to remove some literal dash characters ('-') from several lines before an XML editor I normally use would validate and display the file's content for me to review.

My first question is if I can extract all the location information defined at the lowest hierarchical position along with high level details from the top level xml hierarchical nodes ...

Let me make sure I understand what you want to do. Given a structure like the one illustrated below, you want to bring each <Issue> element in from the <Issues> list as separate events, but you want the events to incorporate the information from the <LocalTime> and/or the <UtcTime> elements found outside the <Issues> structure.

The quick answer is no, you cannot do this. When specifying the 'XML Object Name' parameter for either the 'Poll an External Website for XML' or 'Receive XML on a REST Endpoint' GeoEvent inputs, you are telling GeoEvent that it should scan forward (through the XML structure), skipping past elements and attributes until it locates the specified 'XML Object Name' node. GeoEvent will then bring in elements listed beneath that root node as separate events. The trade-off to identifying a list deep within a structure and having GeoEvent parse that list is that you have told GeoEvent to skip past and disregard elements outside of the list.

You might want to take a look at the discussion thread XML can I use this source? The problem presented in that thread is similar - the date on which each event occurred is outside of the list of events Since each record in the list only specifies a time, you need a way to enrich each of the individual events with the date recorded as an attribute in the list's root node. In your case your challenge is compounded as it looks like each <Issue> can have more than one <Location>.

... I selected 'Location' as my 'XML Object Name'. Can we somehow extract all the location objects with all the elements and attribute information from this xml?

Looking at the XML structure, <Issues> appears to be a list. I would expect I could use GeoEvent to parse the <Issues> structure in order to bring each <Issue> in as a discrete event. Because the various <Location> records are not organized beneath a single root node (e.g. thay are children associated with a particular issue) I don't think the data structure will support GeoEvent bringing each <Location> in as a separate event.

The idea behind the 'XML Object Name' parameter is that you can specify a single node as a "root" node and have GeoEvent only bring in data found beneath that "root" node as discrete (separate) events.

Hope this information is helpful -

RJ