Custom WebSocket Server + StreamLayer + JS API >= 4.9

[Raul_Jimenez]

From this question we realized that the StreamLayer documentation explained that the following could be done:

You may also use your own web socket server, as long as it emits geographic features in the Esri JSON format.

So Jorge Barrachina Gutiérrez and I are just doing an experiment to connect a StreamLayer with a custom websocket server (built with NodeJS) and it's working in JS API 3.x & 4.x except in versions >= 4.9. We noticed in this version (4.9) the StreamLayer was updated, so we guess something might be broken because to our understanding the changes shouldn't affect to the behaviour of this example.

I have recorded a video to show you the issue:

Does anyone have any idea what might be going on? In case you look the source code you can find it on this repo on Github. // cc: John Grayson John Gravois Gavin Rehkemper Sean McGee‌ Max Payson‌ Rene Rubalcava ‌

---

By the way..., sorry for not been at the DevSummit in Palm Springs! :___(, hopefully we'll see some of you at the DevSummit Europe this year!

[JohnGravois]

Hey Raúl Jiménez Ortega‌!

I have no idea how internal changes in 4.9 could be affecting a custom stream service that delivers the same old Esri JSON features, but it definitely looks like you're on to something.

hopefully the folks from the JSAPI team can help out. in the meantime, is your prototype something you can push up on GitHub with instructions to get it up and running?

cc/ Bjorn Svensson‌

[Raul_Jimenez]

Ey John! Thanks! Sure, you can find it here:

GitHub - hhkaos/arcgis_websocket_server: POC of Fake StreamServer 

Best regards!

[mgeorge]

Hi Raul!

In 4.9 the StreamController was completely rewritten for WebGL, and it makes use of additional functionality that is supported by GeoEvent/StreamServices as well as adds support for automatic re-connection. We also now do the feature processing in a worker. With this we can leverage the GPU to render many more features on the screen and handle faster updates, as well as allow users to take advantage of the client-side querying/filters/effects that we have available for feature-like layers. Instead of just adding new features to a global graphics array (which was what the old implementation did and wasn't very performant), we now store features in a client-side spatial index/data-store that is required both for tiled WebGL rendering, as well as client-side querying. This probably makes us more likely to break if we are lacking the information that we expect.

Note that also starting from 4.9 we no longer advertise that users can use their own websocket server, as this might have been a bit of an overstatement to begin with, especially as StreamServices continue to evolve and add new capabilities. Basically this requires faking a StreamService, and if any functionality is missing, the StreamLayer may not function properly. This may be what is happening as this implementation does not appear to include support for setting filters over the websocket connection. We now always send a filter message, even if no filter is present, as this has virtually no overhead and allows for a single code-path. This also effectively tests a connection that would otherwise fail when setting a filter later, though we should add some timeout logic here so we can log errors to make this more apparent. Additionally, it seems like there might be some issues with the service metadata, esp. concerning the links to the available websocket connections (e.g. wss was showing up as being available despite only starting up a ws connection).

Instead, we want to enable developers to be able to just pass in their own websocket connection without having to fake an entire StreamService. Nothing is settled at this point, but we are thinking about exposing the ability for users to pass in a MessagePort that they could then use to process and stream in whatever data they would like. I'm curious to get your thoughts on this and see if this would provide an easier way of accomplishing what you have in mind.

[Raul_Jimenez]

First of all, I would like to apologize to Jorge Barrachina Gutiérrez because in my first message I forgot to attribute recognition for the source code, since he created it. ^_^''

Said that, thanks Matt George for the explanation! 

though we should add some timeout logic here so we can log errors to make this more apparent.

It would be nice yes .

Instead, we want to enable developers to be able to just pass in their own websocket connection without having to fake an entire StreamService. Nothing is settled at this point, but we are thinking about exposing the ability for users to pass in a MessagePort that they could then use to process and stream in whatever data they would like. I'm curious to get your thoughts on this and see if this would provide an easier way of accomplishing what you have in mind.

Yes, well I guess keeping something like the old implementation of the StreamLayer, maybe just renamed as a first approach would be enough at least to start. We would like to have a way to add a layer in a web map receiving data from a custom websocket server, that way we could use it also within an storymap/dashboard/etc.

Cheers!

[Raul_Jimenez]

Finally you made it!!!, "webSocketUrl" is finally here! ^_^ -> StreamLayer | ArcGIS API for JavaScript 4.17 

I have just tested it using your repo and it works , thanks!

[mgeorge]

Yes ... it took us a while but we were finally able get this in! Let me know if you run into any issues.

[Raul_Jimenez]

Thanks Matt George !

I just noticed a weird blinking when the timeInfo was not set properly -> check this video (this is Chrome but It does the same on Firefox).  I first used this code (fields and timeInfo were wrong).

After fixing it is worked as expected ^_^.

Thanks!

P.S: your code also worked

[JorgeBarrachina_Gutiérrez]

No worries Raúl Jiménez Ortega‌ ! Good work!

[AxelleRibault]

Hi,

Following those discussions, is it possible in 4.11 to create a custom Stream Service? or create a Stream Layer with our own websocket connection?

if it is, how do we do that?

Thanks for your help