GeoEvent Processor: Invalid JSON Output on WebSocket

7107
6
09-03-2014 02:20 AM
New Contributor III

The JSON Output via WebSocket on the ArcGIS GeoEvent Extension returns invalid JSON Objects if the GeoEvent Service has to handle more than one track at the same time.

In our test scenario we have a simple GeoEvent Service that receives JSON on a REST endpoint. The output JSON will be provided via WebSocket.

1.png

To simulate the tracks that will be send to the REST endpoint we used a small JavaScript Tool that is attached.

2.png

In general the data that is received via WebSocket contains valid JSON objects, but when sending more than one track to the REST endpoint, we very often receive invalid JSON objects within the data objects of the WebSocket message.

Following is unexpected in the data string of the message:

  • data contains a complete JSON Track followed by one more curly bracket ‘{‘
  • the starting curly bracket ‘{‘ of the JSON Track is missing
  • data contains two complete JSON Tracks without any separator or squared brackets to define it as list  {“geometry”: …. , “TrackID”: 1}{“geometry”: …. , “TrackID”: 2}

It’s possible to see the problem when connection the WebSocket endpoint with the integrated website:

3.png

Is this a already known Issue? Is there any chance to configure the GeoEvent Processor to send exactly one track per message?

The most important thing is that we expect to get a valid JSON object within every WebSocket message.

Reply
0 Kudos
6 Replies
Occasional Contributor

Hi Rainer,

Can you please confirm the version of GeoEvent Processor you are running?

I must admit I have not tested the WebSockets output for GEP, have you tried pushing the JSON to TCP text output and using the TCP utility to compare the two output messages?  This utility is found in the Websockets in GeoEvent Tutorial download package (::WebSockets-10.2.zip\utilities\tcp-console-app),

http://www.arcgis.com/home/item.html?id=3150df0605154a5b99fa365359b6d457

The starting curly brace doesn't appear missing, it is shown on the line above in your image.  Did you expect a comma between each of these unique JSON return messages?

Reply
0 Kudos
Esri Regular Contributor

Hello Rainer -

Yes, please confirm the release of GeoEvent that you are using. I seem to recall occasionally seeing the generic JSON adapter output invalid JSON records when sending many dozens of events a second to an outbound connector. The issue had nothing to do with the TRACK_ID or unique identification of tracks however, so I'm not sure if the issue I'm recalling has anything to do with what you are observing. The issue I'm recalling was addressed in the 10.2.2 product release I believe.

In any case, I used your JavaScript Track Simulator to simulate a half-dozen inbound tracks. Your tool is generating Esri Feature JSON (which GeoEvent interprets / adapts using its "JSON" adapter) as opposed to Generic JSON or the GeoJSON format (supported at the 10.3.1 product release).

I fed your simulator's input into a 'Receive Features on a REST Endpoint' input and directed GeoEvents adapted by that input to a 'Push JSON to an External WebSocket' output.

I created a 'Receive JSON on a WebSocket' input (which runs in SERVER mode) to provide a hosted WebSocket for the output (which runs in CLIENT mode) to connect to. Finally, I created a 'Write to a JSON File', configured to not output formatted JSON, and configured a GeoEvent Service as follows:

Capture.png

I typically refer to this as a "WebSocket Chain". The input to the upper-left, accepting Esri Feature JSON from your simulator, could be receiving any data feed. It sends the GeoEvents it constructs to a WebSocket output which are subsequently received by the paired WebSocket input (the second input illustrated above). GeoEvents received by this second input are then redirected into the system file.

I ran the simulation for five minutes receiving a total of 1788 events on the rest-features-in input. All of the events received by this input were processed through the WebSocket Chain and were written into the *.json system file. Copying the contents of the *.json system file into an on-line JSON validator I found the file's contents to be valid and the expected number of events in the file. If the generic JSON adapter, the component responsible for generating the payload sent out the websocket-external-json-out, were generating invalid JSON I would expect to see some level of event loss in the *.json system file.

I ran my tests using the 10.3.1 release candidate of GeoEvent.

- RJ

Reply
0 Kudos
New Contributor III

Hi,

thank you very much for looking into this issue.

We are using Version 10.2.2 of ArcGIS GeoEvent Extension for Server. We tested this as well in 10.3 and thre problem is still the same...

We are not missing a comma at the shown output. The problem is, that each output burst (at the screenshot three rows are one "burst") has to be valid but it's not.

If you take the example from my screenshot all together and give it to a json validator, it is probably valid. (except of the commata and surrounding square brackets, but this is not the problem).

As we are parsing each "burst"/message on its own, the problem is in my second and third message. In the second message is a curly bracket at the end which makes the message invalid. In the third message is this curly bracket at the beginning missing...

We are facing this problem with our tcp-websocket receiving application and I just used the output of the rest interface to demonstrate the problem. I have not tested the file-out, but I'm pretty sure the problem will be the same. If you validate the file as a hole it will be valid, but if you try to validate each "row" there will be problems...

I hope the problem we are facing is a little bit more precise now and you have maybe some suggestions to circumvent it or fix it at least in 10.3.1.

Reply
0 Kudos
Esri Regular Contributor

Thank you for your clarification Rainer.  I adjusted my test to use an outbound connector configured to use the Generic-JSON adapter and WebSocket Transport running in SERVER mode. This connector configuration was depreciated with the 10.3 release. Event processing throughput is severely impacted when a GeoEvent Service contains an output running in SERVER mode and the GeoEvent Extension is run on multiple machines in an ArcGIS for Server cluster.

The performance considerations, however, are not an issue when running a single instance of the GeoEvent Extension ... and establishing a WebSocket based output running in SERVER mode is the only way to connect a web browser to the WebSocket as a client to review the JSON output in the browser window. I think my test is now closer to replicating your setup.

I was able to reproduce the issue you describe on a 10.2.2 product release after just a few minutes sending 10 events per second. What I'm seeing does look a lot like the issue I was referring to in my previous reply where the generic JSON adapter would occasionally output invalid JSON records. My apologies, I thought this was an issue I observed in 10.2.1 which was fixed in 10.2.2 ... I found the issue record I was recalling and confirmed with a developer that the issue was found in the 10.2.2 release and was addressed in the 10.3 release.

I was able to import my 10.2.2 GeoEvent configuration into a clean 10.3 environment and retest. I used your Track Simulator to simulate 10 separate tracks, pushing 10 events per second into the GeoEvent Service. I ran the simulation for 20 minutes sending over 12,000 events through the GeoEvent Service. Copying the JSON being output to the web browser's window into a text editor for analysis, I could not find any irregularities in the JSON displayed in the web browser client output.

If you continue to observe the issue in either the 10.3 or 10.3.1 product release of GeoEvent, please submit an incident with Esri Technical support and we will work to establish an on-line meeting to reproduce the issue in your environment. We can then work to reproduce it here in Redlands so that we can investigate further.

Best Regards -

RJ

Reply
0 Kudos
New Contributor III

Hi everybody, Hi RJ,

thank you very much for your helpfull answers and sorry very much for my long delay to come back here. We followed your advise and tested everything with 10.3.1 right now. It seems to be that the output is working right now, but actually we cannot really investigate this as we have massive problems with the input right now.

Maybe you can have a look at our input problem as a colleague of mine described it here: GeoEvent Extension 10.3.1 Input (JSON over TCP) stops working if the input receives more than one me...

Thank you very much,

Rainer

Reply
0 Kudos
Esri Regular Contributor

Hello Rainer -

The issue you referred to -- inbound JSON over TCP fails at high velocity when more than one message every 100ms is sent -- is being worked by a developer, Morakot, on the GeoEvent team.

Morakot has been contact with Thorsten Braun from Geosecure as well as Roman Starý from Esri Germany this week (7-Aug 2015). We are aware of the priority with regard to Geosecure's upcoming functional acceptance test and I'm sure that Morakot is making every effort to prepare a fix and deliver something for on-site validation tests.

Best Regards -

RJ

Reply
0 Kudos