topic Re: GeoEvent-Poll an External Website for JSON in ArcGIS GeoEvent Server Questions

GeoEvent-Poll an External Website for JSON

RipaliBatra — Thu, 28 Mar 2024 14:13:51 GMT

I have configured API in the Input connector, but the response of API is not correctly mapped when there is an array or nested JSON.It is only considering the parent JSON and thus all the content is not fetched properly.

Can someone please guide me if I am missing something, or this is how it behaves and there is some other approach to handle it.

API : api.waqi.info/feed/lucknow/?token=fa73462a52dbbb8848d20a0554fb72424af90975

Incorrect JSON:

Re: GeoEvent-Poll an External Website for JSON

Gene_Sipes — Thu, 28 Mar 2024 14:35:15 GMT

Try putting "data" in the JSON object name parameter of the input connector.

Re: GeoEvent-Poll an External Website for JSON

RipaliBatra — Fri, 29 Mar 2024 07:31:04 GMT

Thanks for your reply @Gene_Sipes

I have already configured JSON Object Name with "data" for my case and therefore it is returning the properties correctly which are inside the "data" JSON. But for the nested json the response is not correctly mapped. Eg:- In the below api response sample, properties such as iaqi, attributions are not correctly mapped.

api.waqi.info/feed/lucknow/?token=fa73462a52dbbb8848d20a0554fb72424af90975

Can you please help me on this?

Re: GeoEvent-Poll an External Website for JSON

Gene_Sipes — Fri, 29 Mar 2024 13:31:02 GMT

This might be worth a read, hopefully it helps:
https://community.esri.com/t5/arcgis-geoevent-server-blog/json-data-structures-working-with-hierarchy-and/ba-p/889509#U889509

Also, I have a JSON feed with multi-cardinality structure, albeit a simple one, but in the input definition the property with multi-cardinality looks like that below. Maybe you can adjust your definition similarly. It's the rental uris.

Re: GeoEvent-Poll an External Website for JSON

RipaliBatra — Tue, 16 Apr 2024 15:23:23 GMT

Hi @Gene_Sipes ,

I have validated the multi-cardinality structure, but not sure why nested json properties are still not correctly mapped.

Consider the "attributions" (for example ) in the api response, it does not even map url ,name and logo.I tried multiple ways to map it out but no luck

api.waqi.info/feed/lucknow/?token=fa73462a52dbbb8848d20a0554fb72424af90975

Re: GeoEvent-Poll an External Website for JSON

Gene_Sipes — Wed, 24 Apr 2024 13:54:00 GMT

Hi @RipaliBatra,

Try changing the group cardinality from many to one?

Re: GeoEvent-Poll an External Website for JSON

RJSunderman — Thu, 02 May 2024 01:16:11 GMT

Hello @RipaliBatra --

Given the rich hierarchical structure of the JSON you are receiving from the https://waqi.info web service you really cannot rely on the GeoEvent Sampler for a good representation of the data. The sampler struggles when given complex hierarchical JSON.

A better approach is to create one or more Write to a JSON File outputs and use them to log the event records emitted from different processors along your configured event processing workflow.

I have a GeoEvent Server 11.1 deployment that I used to test your feed. I was able to allow a Receive JSON on a REST Endpoint input to generate a GeoEvent Definition for me. A sample of the data I sent to my GeoEvent Server input and the generated GeoEvent Definition are shown below {Fig 1] and [Fig 2]. Note that I specified the input use data as the root of the JSON structure when adapting the received JSON.

Note: Be careful when relying on auto-generated GeoEvent Definitions. An input will make its "best guess" as to what the GeoEvent Definition ought to be based on the first data record it receives. But the generated event definition will often use Double for data values received as epoch long integer values (for example). You have to review the generated GeoEvent Definition and verify that each array and element will be adapted properly for the data you expect to receive.

I was able to configure a Field Mapper to pull specific values out of the hierarchical JSON:

Note the expressions being used to access the data:

Attributions is an array, so we have to provide the index of the element we want to access from that array. attributions[1].name accesses the second element in the array, the one with the named string "World Air Quality Index Project".
City is the name of an element, so we do not access it with ordinal values like we do when accessing the array. city.name is sufficient to retrieve the string "B R Ambedkar University, Lucknow, India".
The latitude and longitude coordinates in the city element, however, are in an array nested within an element. When pulling these as separate values we have to specify their ordinal positions in the array:
- The latitude is city.geo[0]
- The longitude is city.geo[1]
The iaqi values are grouped within an element. Accessing them is fairly straightforward:
- iaqi.co.v
- iaqi.no2.v
- iaqi.pm10.v

In my Field Mapper illustration I only pulled the string for the "day", but if we wanted to be a little more creative we could build a string from available values for a given day. The string value "Day: 2024-05-01 (Avg/Min/Max: 29.0 / 14.0 / 50.0)" could be build using the following expression to pull individual values and append them together -- taking care to use the toString( ) function to explicitly cast Double values to String values when appending them to literal strings:

'Day: ' + toString(forecast.daily.o3[2].day) + ' (Avg/Min/Max: ' + toString(forecast.daily.o3[2].avg) + ' / ' + toString(forecast.daily.o3[2].min) + ' / ' + toString(forecast.daily.o3[2].max) +')'

Because the daily forecast values for "o3", "pm10", and "pm25" all have the same elemental structure you will not be able to use a Multicardinal Field Splitter processor to collapse or flatten these three arrays. It looks like these arrays are allowed to contain a variable number of items. The array o3 has 8 elements whereas the arrays pm10 and pm25 both have 9 elements. There is no iterator or looping mechanism available in any of GeoEvent Server's processors, so you will very likely have to stick with extracting essential information from the JSON using field calculation expressions like I show above.

Hope this information helps, and a special Thank You to @Gene_Sipes for jumping in to help with this complicated hierarchical JSON.

-- RJ

Re: GeoEvent-Poll an External Website for JSON

Gene_Sipes — Thu, 02 May 2024 13:30:03 GMT

Always love your explainers! Thanks for jumping in @RJSunderman