aversteresriaustralia-com-au-esridist

Configuring an XML Receiver Input in GeoEvent Server

Blog Post created by aversteresriaustralia-com-au-esridist Employee on Mar 16, 2020

GeoEvent Server is a powerful piece of software when it comes to ingesting and processing incoming data, especially when the data is not in a format, compatible with ArcGIS.

 

Recently, we came across a scenario where the GPS positions of 2-way radios was to be ingested and displayed in ArcGIS. The data from the radio environment could only send data to a URL and could only send XML string in a certain structure. The client did not have the capability of altering the XML strings structure or format.

To comply with these requirements, GeoEvent Server was identified as the suitable ingestion and processing tool.

 

The greatest challenge was how to use the incoming XML string and process so that it could be stored and accessed in ArcGIS Portal.

 

The string looked something like this:

 

<?xml version="1.0" encoding="utf-16"?>

<data>

                <source>[address insert here]</source>

<device>

                <Id>1234</Id>

</device>

<update>

                <time>20191014T060927.557+0000</time>

                <field>

                                <id>1</id>

                                <dtype>float</dtype>

                                <value>-33.100358</value>

                </field>

                <field>

                                <id>2</id>

                                <dtype>float</dtype>

                                <value>150.842401 </value>

                </field>

                <field>

                                <id>3</id>

                                <dtype>int</dtype>

                                <value>3</value>

                </field>

                <field>

                                <id>4</id>

                                <dtype>int</dtype>

                                <value>1</value>

                </field>

                <field>

                                <id>5</id>

                                <dtype>boolean</dtype>

                                <value>0</value>

                </field>

                <field>

                                <id>6</id>

                                <dtype>string</dtype>

                                <value>345</value>

                </field>

</update>

</data>

The parameters or data fields are contained within the XML document format:

 

Parameter

Description

XML element name

Time Stamp

The time stamp associated with the position

<time>

Source Identifier

Machine IP address of source

<source>

Device Identifier

Unique identifier of the device

<id>

Field

Defines a field, which represents a specific data entry. Each update contains multiple fields.

<field>

Field Identifier

Unique identifier for a field.

<id>

Field Data Type

Data Type (float, int, boolean, string, double)

<dtype>

Field Data Value

The field value

<value>

 

And the field identifiers are defined as follows:

 

Field ID <id>

Field Data type <dtype>

Field Definition

1

Float

GPS Latitude

2

Float

GPS Longitude

3

Integer

GPS Speed

4

Integer

GPS Direction

5

Boolean

PTT Active

6

String

Call ID

 

 

How to create the Input.

 

  1. In the Inputs tab under Services, configure a Receive XML on a REST Endpoint input and allowed it to create a GeoEvent Definition. Named this initial event definition XML_Receiver_Initial and confirmed you can HTTP/POST the block of XML to the GeoEvent Server input’s endpoint.

 

It is important at this point that you HTTP/POST the block of XML to the GeoEvent Server input’s endpoint in order for GeoEvent Server to generate the definition schema, which we will use in the future steps.

  1.      Navigate to Site and make a copy of the GeoEvent Definition input created, naming the copy XML_Receiver_Final and changed field from cardinality ‘One’ to cardinality ‘Many’.

Leave the fields in the definition of the data types set to String. All JSON comes in as string values, and it is assumed that XML is the same. There are 2 fields that are floating point values, which can be mapped to a field whose type is Double. This will be addressed later.

 

            Save it.

 

  1.       Next, edit the input to stop assuming it is allowed to create a GeoEvent Definition and configured it to use the XML_Receiver_Final event definition tailored for this data feed. Also configure the input to start considering data as an XML Root Node.

 

Make sure nothing done so far has affected the ability to HTTP/POST, receive the XML, and successfully adapt it to create a GeoEvent event record.

 

Once it has been established that the Input is still receiving the records successfully, we must think about what the desired data structure in the ArcGIS environment would look like and build a GeoEvent Definition to reflect this.

The attributes we want to keep are the IP address, reported date/time, Latitude, Longitude, the Call ID, Device Status and the Geometry.

 

  1.       In the Site tab,create a GeoEvent Definition named EventRecord-string. All fields will remain string for the time being while we will also add a geometry field.

            Save it.

 

  1.       In the Input connector, options are available to create a Geometry from the incoming string. We will create this Geometry field now by calling <field><id> 1 and 0 (Lat and Long) in the Input connector’s Advanced configuration. Toggle the Construct Geometry from field value to Yes. The mapping configuration looks like this:
  •       update.field[1].value mapped to Latitude
  •       update.field[0].value mapped to Longitude

 

  1.       Next thing to do is to wire up the input to the output and pass the event records through a Field Mapper, just to see if it is able to pull values assumed to be longitude and latitude out of the received data, which has already undergone adaptation and translation by the inbound connector. id can write it into DeviceId, likewise source transfers directly to IP_Address. Both are string-to-string mappings, which is preferable.

       The date/time will take some more effort and will be addressed later.

 

  1.       Configure a Write to a JSON file output and designate a location on your local machine for the file to write to, that is accessible by GeoEvent Server. Since the purpose of the file is to test the Input, we will not spend much time on Outputs.

 

  1.       Create a GeoEvent Service, drag both the XML_Receiver Input and file-json-out-full Output into the window.

 

   9.  Create a Processor that will map the Received Schema Definition to the Output Definition. Here we will be keeping          all incoming fields in String for the time being.

      10. Connect the items in the window and publish the service. Name it geoevent_gps_positions.

 

    11. When successfully Published and the service has Started, HTTP/POST the XML to the Input connector’s URL to           test if an output is generated.

 

If this has been successfully tested, and the JSON file has the content, then we will proceed to extract and process the data, to make it useful to us.

 

  1.       Next we will address the date/time field. We have deliberately ingested its value as a String because it is not  formatted for GeoEvent’s input to adapt to a Date without some nasty configuration.

 

  1.       Navigate to the GeoEvent Services window and open you service. Add a Processor to the window. We will use a Field Calculator with the nasty bit of RegEx to reformat the string from 20191014T060927.557+0000 to 2019-10-14T06:09:27.557Z:

   

replaceAll(ReportedDT_String, '^(\d{4})(\d{2})(\d{2})[T](\d{2})(\d{2})(\d{2}).*', '$1-$2-$3T$4:$5:$6')

 

 

            Publish the service. It is fine to leave the items unconnected for the moment.

 

At this point, we have a processor that reads the incoming data and writes the values as string into an output schema definition. We then built a processor that format date, which is still a string datatype. Next, we have to write all the value to a definition that complies with the value’s true datatypes.

 

  1.       Navigate to the Site Copy the previously created EventRecord-string definition and rename it EventRecord-Final.

 

  1.       Edit each field to comply with its correct datatype as per the above table.

 

Add another Date field for the calculated ReportedDT_String.

Now that the final GeoEvent Schema Definition has been created, we can write the incoming XML string into the desired table structure, complying with the specific datatype requirements.

 

  1.      Navigate back to Service>GeoEvent Services and add another Processor.

      Configure another Field Mapper with the Source GeoEvent Definition, the EventRecord-string and the Target       Geoevent Definition the EventRecord-Final.

 

Note the ReportedDT_String field is called to populate the new date field reporteddt.

 

Evolving the solution to use two Field Mappers gives you the freedom to have the Field Calculator write its result into a String field and review the produced result by routing the event record(s) to the JSON File output.

 

One important limitation to recognize is that none of the existing processors can write to a hierarchical, multicardinal data structure. They can read from one, but not write to one. If we were to try using a Field Calculator to write to a structure that was not pre-flattened, all the data in nested hierarchical attributes will come out as null. So, this really is the best time to evolve the solution to use two Field Mappers.

 

A second important bit you want to remember is that you do not want to write, or field map the date/time string value to a Date until you’re absolutely sure you’ve got the ISO 8601 formatting correct. That is the only format, by the way, you can use express a date/time value as a String and have a Field Mapper or Field Calculator handle the implicit cast to Date.

 

I hope this helps anyone with the same or similar issues.

 

Special thanks to RJ Sunderman for his help with this solution.

Outcomes