Watch a Folder for New JSON Files generates a GeoEventDefinition with missing attributs

[kavi88]

Hi guys,

I am using the "Watch a Folder for New JSON Files" as an ArcGIS Geoevent input connector, however, the GeoEventDefinition is being created with some missing attributes.

The json file looks like this :

{ "application/json": [ { "nomElement": "", "Description": "", "attributs": [], "elements": [ { "nomElement": "", "Description": "", "attributs": [], "elements": [] }, { "nomElement": "", "Description": "", "attributs": [ { "nomAttribut": "", "values": [ { "mesure": "", "horodatage": "", "unite": "" } ] }, { "nomAttribut": "", "values": [ { "mesure": "", "horodatage": "2021-10-05T14:23:44.1376727+02:00", "unite": "" } ] } ], "elements": [] },

The created definition looks like this:

While the input connector configuration looks like this :

I am wondering if i am missing any thing ? @RJSunderman @EricIronside 

Thanks in advance.

[RJSunderman]

Hello Suzy --

The hierarchy in the JSON data structure you illustrated looks almost recursive. If this is a typical example of the JSON an input you have configured would expect to receive as its first data record, I think the inbound adapter is making its best guess at what the GeoEvent Definition ought to be. An inbound adapter's guess will be more accurate for simpler data structures -- the adapter will not recursively iterate through a data structure to further refine what it sees up-front.

For example, in my illustration above I've added just a little white-space and formatting to your JSON example. The line I've designated "Ex. 01" in green is intended to be an array capable of holding zero or more data values. But the adapter, given the empty array in the illustration, does not know whether the array's data will (eventually) be a set of integer values, a set of string values, or a set of JSON elements with a more detailed sub-structure.

Given the empty array, the inbound adapter makes a guess and assumes a data type of String and a cardinality for the data value of Many (indicated with the infinity symbol circled in green in the illustrated GeoEvent Definition). As long as the data values eventually received in that array can be implicitly cast to String the adapter will be able to parse and adapt data it (eventually) receives.

Looking at another part of the data structure, designated "Ex. 02" and highlighted in orange, we see an "elements" array which contains JSON elements (as opposed to primitive String or Integer values). But the two JSON elements shown in the example have potentially different sub-structures.

The first "attributes" array (boxed off in orange) is empty. As before the inbound adapter makes a guess and assumes an eventual data type of String setting the cardinality to Many (result circled in orange in the illustrated GeoEvent Definition). Again, the inbound adapter is not going to recuse more deeply into the data to see the other "attributes" (boxed off in blue) -- so it will not see that its first assumption is wrong and that "attributs" is probably not going to be an array of String values. The inbound adapter will not know that a key "attributes" found within "elements" is actually be intended to hold zero of more JSON elements with their own sub-structure.

A quick test, removing the line I've designated "Ex. 02" above, confirms that a JSON inbound adapter receiving the JSON illustrated below will assume a data type Group for an "attributs" key found nested beneath a key "elements":

 
I suspect the GeoEvent Definition in this second illustration is closer to what you were expecting.

Hope this information helps --
RJ

[kavi88]

Hey @RJSunderman 

Thank you for the detailed answer, indeed the GeoEvent Definition in this second illustration is closer to what i was expecting for now, but i am still wondering if there is another way that can make me have the exact same structure in the definition as later the same json structure will used in an api and i would like to use the "Poll an external website for JSON as an input , and eventually have all the coming data.

Thank you in advance.

Suzy

[RJSunderman · Accepted solution]

Hey Suzy --

Because JSON is self-describing and provides names for all the key/value pairs in a data structure, a input you configure is able to ignore "missing" data and handle the case when "extra" data arrives which is not represented in a GeoEvent Definition. Stated another way, if data received does not contain a particular key/value represented in the GeoEvent Definition being used to adapt that data, the attribute value in the adapted event record will be assigned a null value. If a received data record contains a key/value which is not represented in the GeoEvent Definition then the extra data will be ignored.

You do not need to allow an input or inbound connector to create a GeoEvent Definition for you. In fact, after sending a bit of sample data to an input, so its adapter can create a GeoEvent Definition for you, the recommended best practice is to copy the auto-generated definition and make whatever edits are needed. You might want, for example, to handle a particular data value as a Date rather than a Double or handle a numeric value as a Long even though it is sent to you as a quoted String value.

You then delete the auto-generated GeoEvent Definition and reconfigure your input to use the GeoEvent Definition you have reviewed and edited.

As for integrating with an external API, I would assume you would first identify the REST web service endpoints you want to poll for data and obtain a sample of the data to help you administratively create a GeoEvent Definition which reflects the expected data structure. You can then configure an input to use that GeoEvent Definition as it adapts data records it receives in response to data requests it sends. (If you are asking for an automated way of creating data structures and updating them on-the-fly as new data feeds are discovered, I'm sorry, but I don't have a good solution for that.)

Normally a data provider will provide a specification from which you can create a GeoEvent Definition. Sometimes the data provider will document an actual sample of their data -- and that is when you can allow your input to (temporarily) create a GeoEvent Definition for you based on the first data record it receives. But you will always want to copy, edit, and then delete the auto-generated event definition as a best practice. You can always delete an auto-generated GeoEvent Definition and allow the input to poll again to see if/how a feed's data has changed so that you can make adjustments to an authoritative GeoEvent Definition you are developing for a particular data feed. There is a "Learning Mode" that you can toggle 'on' to briefly give the inbound connector permission to adjust a GeoEvent Definition it has created based on variable data structures it receives -- but the adapter is only going to add newly observed key/value pairs as named attributes to an existing GeoEvent Definition to try and accommodate data it is just now seeing that it didn't see before. Even with "Learning Mode" toggled on the adapter is not going to recursively iterate through a received data structure.

In the case we were discussing, if at one time the adapter saw an empty array and decided to handle/adapt data for that attribute as an array of String values, it will not change that attribute specification based on data seen later (which might be a collection of JSON elements with a specific sub-structure) to update the GeoEvent Definition and change String:Many to be Group:Many. The adapter will simply fail to adapt the array of JSON elements as primitive String values. You will have to administratively adjust the GeoEvent Definition the adapter is using to accommodate the type of data elements the data feed is actually placing into the now populated array.

-- RJ