topic Re: Process JSON Collection? in ArcGIS GeoEvent Server Questions

Process JSON Collection?

DennisGeasan — Fri, 13 Dec 2013 00:04:17 GMT

Anyone know how I can get the GEP to handle a JSON collection? I want these to feed into a feature service that includes a point feature class.
DG

{
    "lastLocations" : [{
            "driverID" : 83886089,
            "driverEmployeeId" : null,
            "vehicleID" : 83886595,
            "vehicleName" : "PVLV1139",
            "head" : 5,
            "speed" : 0,
            "time" : "2013-08-07T13:33:01Z",
            "location" : {
                "lat" : 45.443727,
                "lng" : -129.978607
            }
        }, {
            "driverID" : 83886089,
            "driverEmployeeId" : null,
            "vehicleID" : 83886596,
            "vehicleName" : "PVLV1219",
            "head" : 3,
            "speed" : 0,
            "time" : "2013-10-03T00:14:50Z",
            "location" : {
                "lat" : 41.920233,
                "lng" : -175.161255
            }
        }
    ]
}

Re: Process JSON Collection?

RyanElliott — Fri, 13 Dec 2013 15:18:23 GMT

GEP has two built in JSON parsers, designed to handle "JSON Features" and "Generic JSON". The JSON structure you describe can be parsed by the generic JSON adapter.

In your example, there are two vehicle reports, so you probably want two separate GeoEvents to be generated from that data. If you set the adapter's property called "JSON Object Name" to "lastLocations", then the parser will find that key in the JSON Structure, notice that it is an array, and turn each object in that array into a GeoEvent.

The data looks pretty basic, so you can tell the adapter to "Create GeoEvent Definition" and it will install a new GeoEvent Definition with the name you specify by examining the incoming data and inferring field names and types. It's always good to review the GeoEvent Definition to make sure the field's are the right types (e.g. Dates were correctly identified). Also, you should apply a TRACK_ID tag to the appropriate field.

You can also tell the adapter to "Construct Geometry from Fields", and you will be asked for the X, Y, Z, fields. Set the X property to "location.lng" and the Y property to "location.lat". The other geometry-related fields can be left blank.

Re: Process JSON Collection?

DennisGeasan — Fri, 13 Dec 2013 15:36:53 GMT

I have already done as you suggest, sort of. I set "JSON Object Name" to "location" not "lastLocations". However, the only attributes I get are the 'lat' and 'lng' fields and the constructed geometry. I want all the attributes of each collection object and that each object be an event directed to a record in a point feature class. I have tried to set "JSON Object Name" to "lastLocations" but then the input adapter fails to collect anything.

I'm thinking I will have to reformat and flatten the JSON before feeding it into the GEP.

For version 10.2, if I set "JSON Object Name" to "location", then the output is the two geoevents as defined in the JSON source. If I don't define a "JSON Object Name" then it seems the whole collection is treated as one event rather than multiple events. Not sure if that is a feature or bug.

DG

Re: Process JSON Collection?

RJSunderman — Fri, 13 Dec 2013 15:39:14 GMT

Hey Dennis -

Attached is an illustration from the 10.2.0 product, supporting what Ryan says above.
The interface panels change slightly at 10.2.1 ... but not so much that you can't use what is below.

[ATTACH=CONFIG]29862[/ATTACH]

- RJ

Re: Process JSON Collection?

RyanElliott — Fri, 13 Dec 2013 16:01:18 GMT

I have tried to set "JSON Object Name" to "lastLocations" but then the input adapter fails to collect anything.

Are there any errors in the log files? Also did you let the adapter create the GeoEvent Definition? You may want to stop the input, delete the GeoEvent Definition, and then have the adapter recreate it (especially if you are changing other settings).

For version 10.2, if I set "JSON Object Name" to "location", then the output is the two geoevents as defined in the JSON source. If I don't define a "JSON Object Name" then it seems the whole collection is treated as one event rather than multiple events. Not sure if that is a feature or bug.

Yes, that is expected behavior.

Re: Process JSON Collection?

DennisGeasan — Fri, 13 Dec 2013 16:06:29 GMT

Full stop!!! I went back and tried again setting "JSON Object Name" to "lastLocations" and defining the fields to build the geometry to location.lng and location.lat. Now it appears to be working!! Hmm. Obviously I did something dumb early on and should have retraced my earlier steps. Thanks for the help. I'm on my way.
DG

Re: Process JSON Collection?

JamesStewart2 — Fri, 11 Dec 2015 20:05:17 GMT

I know this thread is 2 years old, but I'm facing the same issue and I can't get the geometry data into the output feature class. This is some json output straight from the input connector:

The data is being delivered in an array called "Updates". When I add "Updates" to the JSON Object Name setting in the input connector, the data is parsed into separate geoevents. At this point, I can get all the attributes into the feature class except for Geometry. That shows up in the new GeoEvent definition as a group.

This is the output json data after adding "Updates" to the JSON Object Name setting in the input connector:

This is the new GeoEvent definition showing the grouped geometry attributes:

I tried the 'Construct Geometry from Fields' option as shown below...

...but when I did, the input connector was no longer streaming any data through. I have tried various processor options to construct a new geometry field including the Projector processor. Nothing works. Can anyone tell me what I'm missing?

Re: Process JSON Collection?

RJSunderman — Sun, 31 Jan 2016 03:29:54 GMT

Hello James -

I tried the 'Construct Geometry from Fields' option as shown below...
...but when I did, the input connector was no longer streaming any data through.

I think the reason that the input stopped showing data records being received (when you toggled its 'Construct Geometry from Fields' to 'Yes') is that the GeoEvent structure already has a field named 'Geometry' which is preventing the input's adapter from creating a field it wants to name 'Geometry' into which it can place the value it has created.

Assuming that you have no influence over the data provider's choice of naming the group element, here's something you can do to work around the issue.

Bring the event data in, as provided, without attempting to build a Geometry
Use a 'Field Mapper' processor to remap the event structure:
- Geometry.X => Longitude
- Geometry.Y => Latitude
- Geometry.spatialReference.wkid => WKID
Send the remapped event to a 'Push JSON to an External Website' output
- The 'URL' parameter will be the REST endpoint of a new 'Receive JSON on a REST Endpoint' input you create
- e.g. http://mymachine.mydomain.com:6180/geoevent/rest/receiver/rest-json-in-repost
Configure the rest-json-in-repost input you create to construct a Geometry from the Latitude, Longitude, and WKID field values

When configuring the 'Push JSON to an External Website' output, I elected to clear the parameter field 'Header Parameter Name:Value List' (since I wasn't actually providing a list of name:value pairs...)

What I ended up doing was placing both the rest-json-in and rest-json-in-repost inputs in the same GeoEvent Service. This might lead to some confusion with your event counts on the Monitor page as every event received increments the GeoEvent Service's event count by 2 (since it has two inputs and is essentially sending a copy of the event record to itself).

My solution uses three GeoEvent Definitions ... an original matching the data provider's structure, a remap of that structure, and a copy of the remapped structure with a Geometry field added (used by the rest-json-in-repost input constructing the Geometry, so it has somewhere to place the Geometry it constructs).

Please let me know how this works for you -

Re: Process JSON Collection?

JamesStewart2 — Wed, 03 Feb 2016 22:59:44 GMT

That's an interesting solution that didn't occur to me. What I ended up doing is similar and uses two GeoEvent services. The first one uses Field Calculator processors to calc the geometry data to the fields longitude, latitude, and wkid. Then there's a Field Reducer that strips out the Geometry attribute and the resulting stream is output to cache. The second service inputs from the cache, creates the geometry from the longitude, latitude and wkid fields and outputs to the appropriate service. I'd prefer to have only one GeoEvent service, but this works. I'm gonna give your approach a try just to check it out, but what I've done is working and that's what I needed. Thanks for looking into this for me.