Geofeedia's new api as input into feature service output

NathanEnge · ‎12-09-2013

sorry but I'm a newb
I have a query string from Geofeedia's new api:
https://api.geofeedia.com/v1/search/geofeed/18591?appID=fbf45924&appKey=23ced6fac3544364b8c4b84aee76...

Currently they only have this json and the geofeedrss. After running the above URL through JSON pretty, I can see that I don't need the pre-amble header information - just each entry within the "items" tag.

I've tried just a simple JSOn input, and CSV output with just the default connector and all I get is the header as an entry.
When I try to fieldmap I get only two definitions - the default "incident" and the definition created by the input. However, the headings that I want aren't there. I'm assuming I need to create a new definition, and just grab the tags I need:

"items": [
    {      "latitude": -22.75003063,      "longitude": -43.46636981,      "source": "twitter",      "externalId": "410192177298681856",      "publishDate": "2013-12-09T23:40:10Z",      "createdOn": "2013-12-09T23:40:42Z",      "title": "@annaclaarafrc nossa que coincid\u00eancia anna clara",      "description": "",      "author": {        "name": "Pehmota",        "url": "http:\/\/twitter.com\/Pehmota",        "avatar": {          "url": "https:\/\/pbs.twimg.com\/profile_images\/378800000435495804\/4dac428b3e3c4d1e568da73bf6ce96c0_normal.jpeg"        }      },      "url": "http:\/\/twitter.com\/Pehmota\/status\/410192177298681856",      "mediaItems": [              ]    },    {      "latitude": -22.88979542,      "longitude": -43.27738588,      "source": "twitter",      "externalId": "410192176875053056",      "publishDate": "2013-12-09T23:40:10Z",      "createdOn": "2013-12-09T23:40:42Z",      "title": "HAHAAAAAA VAI TOMAR NO CU TODO MUNDOOOOOOOOOO KKKKKKKKKKK",      "description": "",      "author": {        "name": "RodriguesPerseu",        "url": "http:\/\/twitter.com\/RodriguesPerseu",        "avatar": {          "url": "https:\/\/pbs.twimg.com\/profile_images\/378800000822948774\/b233ee2ea64ecb75a60e059d4910e98d_normal.jpeg"        }      },      "url": "http:\/\/twitter.com\/RodriguesPerseu\/status\/410192176875053056",      "mediaItems": [              ]     },

ultimately I want to turn the geofeedia into a live feature service to consume in AGS.

RJSunderman · ‎12-09-2013

Hello Nathan -

I've tried just a simple JSON input and CSV output with just the default connector and all I get is the header as an entry.

I wouldn't recommend a CSV output for this data. When I created an 'Poll an external website for JSON' Input connector and allowed GEP to generate a GeoEvent Definition for me, the first thing I noticed was the complexity of the data. Data with groups and lists (elements whose cardinality is greater-than one) cannot be represented as comma separated text. The data would be ambiguous; you wouldn't know if an item after a comma were the next element, the next sub-element within a group, or the next item in a list.

That aside, I found that I was receiving INFO and ERROR messages in the GeoEvent Processor logfile for the https://api.geofeedia.com/v1/search/geofeed/18591?appID=fbf45924&appKey=23ced6fac3544364b8c4b84aee76... data feed. When my Input connector first polls for data, I see an ERROR message which looks like (click to enlarge):
[ATTACH=CONFIG]29698[/ATTACH]

I am able to use JSONLint to parse the feed's JSON, but I think that GeoEvent Processor is having a problem with the JSON structure. When I stop and restart the Input connector, I will see a message from com.esri.ges.messaging.jms.JmsMessaging saying "Error trying to receive message. Output may not have processed message". This is just an INFO message, but then when the Input next polls the external site, the error in the screenshot above is logged.

If I configure the Input connector's 'JSON Object Name' property to look for "items" (no quotes), I get a simpler GeoEvent Definition:
[ATTACH=CONFIG]29699[/ATTACH]

But only a handful (I was only seeing 11) events get output to the JSON file being managed by the Output connector. The JSON in the file doesn't appear valid at first glance, the GEP Monitor continues to increment the count for the Output connector, but no additional events get written to the output. The GEP logfile is full of messages from com.esri.ges.messaging.jms.JmsMessaging saying "Error trying to receive message. Output may not have processed message".

I will look into this further - but I wanted to get you my initial recommendation (not to use a CSV file for output) and record my initial observations in case they are at all helpful to you.

- RJ

NathanEnge · ‎12-10-2013

RajI do know the API can output in georss and I have been successful adding the feed to agol. The structure is cleaner. I could explore the georss to feature service route as we'll.

Hello Nathan -I wouldn't recommend a CSV output for this data. When I created an 'Poll an external website for JSON' Input connector and allowed GEP to generate a GeoEvent Definition for me, the first thing I noticed was the complexity of the data. Data with groups and lists (elements whose cardinality is greater-than one) cannot be represented as comma separated text. The data would be ambiguous; you wouldn't know if an item after a comma were the next element, the next sub-element within a group, or the next item in a list.That aside, I found that I was receiving INFO and ERROR messages in the GeoEvent Processor logfile for the https://api.geofeedia.com/v1/search/geofeed/18591?appID=fbf45924&appKey=23ced6fac3544364b8c4b84aee76... data feed. When my Input connector first polls for data, I see an ERROR message which looks like (click to enlarge):[ATTACH=CONFIG]29698[/ATTACH]I am able to use JSONLint to parse the feed's JSON, but I think that GeoEvent Processor is having a problem with the JSON structure. When I stop and restart the Input connector, I will see a message from com.esri.ges.messaging.jms.JmsMessaging saying "Error trying to receive message. Output may not have processed message". This is just an INFO message, but then when the Input next polls the external site, the error in the screenshot above is logged.If I configure the Input connector's 'JSON Object Name' property to look for "items" (no quotes), I get a simpler GeoEvent Definition:[ATTACH=CONFIG]29699[/ATTACH]But only a handful (I was only seeing 11) events get output to the JSON file being managed by the Output connector. The JSON in the file doesn't appear valid at first glance, the GEP Monitor continues to increment the count for the Output connector, but no additional events get written to the output. The GEP logfile is full of messages from com.esri.ges.messaging.jms.JmsMessaging saying "Error trying to receive message. Output may not have processed message".I will look into this further - but I wanted to get you my initial recommendation (not to use a CSV file for output) and record my initial observations in case they are at all helpful to you.- RJ

RJSunderman · ‎12-12-2013

Hello Nathan �??

I�??ve been working with Scott Mitchell over at GeoFeedia and investigating your issue with using GeoEvent Processor to consume the JSON returned when you poll GeoFeedia�??s site. I wanted to share with you a few observations / discoveries.

1) As I said mentioned above, data with groups and lists (elements whose cardinality is greater-than one) cannot be represented as comma separated text, so you will want to use an Output �??Write to a .json file�?? rather than writing to a CSV file. Writing the data as comma separated text would be ambiguous. You wouldn't know if an item after a comma were the next element, the next sub-element within a group, or the next item in a list.

2) You can use the URL you shared with me, Product" rel="nofollow" target="_blank">http://www.arcgis.com/home/group.html?owner=GeoEventTeam&tit... Gallery helpful in resolving your issue with getting the GeoFeedia data into a feature service.

Hope this information helps �??
RJ