topic Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9 in ArcGIS JavaScript Maps SDK Questions

Custom WebSocket Server + StreamLayer + JS API >= 4.9

Raul_Jimenez — Tue, 05 Mar 2019 13:03:24 GMT

From this question we realized that the StreamLayer documentation explained that the following could be done:

You may also use your own web socket server, as long as it emits geographic features in the Esri JSON format.

So Jorge Barrachina Gutiérrez and I are just doing an experiment to connect a StreamLayer with a custom websocket server (built with NodeJS) and it's working in JS API 3.x & 4.x except in versions >= 4.9. We noticed in this version (4.9) the StreamLayer was updated, so we guess something might be broken because to our understanding the changes shouldn't affect to the behaviour of this example.

I have recorded a video to show you the issue:

Does anyone have any idea what might be going on? In case you look the source code you can find it on this repo on Github. // cc: John Grayson John Gravois Gavin Rehkemper Sean McGee‌ Max Payson‌ Rene Rubalcava ‌

---

By the way..., sorry for not been at the DevSummit in Palm Springs! :___(, hopefully we'll see some of you at the DevSummit Europe this year!

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

JohnGravois — Thu, 07 Mar 2019 23:09:22 GMT

Hey Raúl Jiménez Ortega‌!

I have no idea how internal changes in 4.9 could be affecting a custom stream service that delivers the same old Esri JSON features, but it definitely looks like you're on to something.

hopefully the folks from the JSAPI team can help out. in the meantime, is your prototype something you can push up on GitHub with instructions to get it up and running?

cc/ Bjorn Svensson‌

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

Raul_Jimenez — Fri, 08 Mar 2019 05:20:40 GMT

Ey John! Thanks! Sure, you can find it here:

GitHub - hhkaos/arcgis_websocket_server: POC of Fake StreamServer

Best regards!

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

mgeorge — Fri, 08 Mar 2019 22:04:09 GMT

Hi Raul!

In 4.9 the StreamController was completely rewritten for WebGL, and it makes use of additional functionality that is supported by GeoEvent/StreamServices as well as adds support for automatic re-connection. We also now do the feature processing in a worker. With this we can leverage the GPU to render many more features on the screen and handle faster updates, as well as allow users to take advantage of the client-side querying/filters/effects that we have available for feature-like layers. Instead of just adding new features to a global graphics array (which was what the old implementation did and wasn't very performant), we now store features in a client-side spatial index/data-store that is required both for tiled WebGL rendering, as well as client-side querying. This probably makes us more likely to break if we are lacking the information that we expect.

Note that also starting from 4.9 we no longer advertise that users can use their own websocket server, as this might have been a bit of an overstatement to begin with, especially as StreamServices continue to evolve and add new capabilities. Basically this requires faking a StreamService, and if any functionality is missing, the StreamLayer may not function properly. This may be what is happening as this implementation does not appear to include support for setting filters over the websocket connection. We now always send a filter message, even if no filter is present, as this has virtually no overhead and allows for a single code-path. This also effectively tests a connection that would otherwise fail when setting a filter later, though we should add some timeout logic here so we can log errors to make this more apparent. Additionally, it seems like there might be some issues with the service metadata, esp. concerning the links to the available websocket connections (e.g. wss was showing up as being available despite only starting up a ws connection).

Instead, we want to enable developers to be able to just pass in their own websocket connection without having to fake an entire StreamService. Nothing is settled at this point, but we are thinking about exposing the ability for users to pass in a MessagePort that they could then use to process and stream in whatever data they would like. I'm curious to get your thoughts on this and see if this would provide an easier way of accomplishing what you have in mind.

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

Raul_Jimenez — Tue, 12 Mar 2019 09:29:22 GMT

First of all, I would like to apologize to Jorge Barrachina Gutiérrez because in my first message I forgot to attribute recognition for the source code, since he created it. ^_^''

Said that, thanks Matt George for the explanation!

though we should add some timeout logic here so we can log errors to make this more apparent.

It would be nice yes .

Instead, we want to enable developers to be able to just pass in their own websocket connection without having to fake an entire StreamService. Nothing is settled at this point, but we are thinking about exposing the ability for users to pass in a MessagePort that they could then use to process and stream in whatever data they would like. I'm curious to get your thoughts on this and see if this would provide an easier way of accomplishing what you have in mind.

Yes, well I guess keeping something like the old implementation of the StreamLayer, maybe just renamed as a first approach would be enough at least to start. We would like to have a way to add a layer in a web map receiving data from a custom websocket server, that way we could use it also within an storymap/dashboard/etc.

Cheers!

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

JorgeBarrachina_Gutiérrez — Tue, 12 Mar 2019 09:48:51 GMT

No worries Raúl Jiménez Ortega‌ ! Good work!

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

AxelleRibault — Tue, 23 Apr 2019 13:24:15 GMT

Hi,

Following those discussions, is it possible in 4.11 to create a custom Stream Service? or create a Stream Layer with our own websocket connection?

if it is, how do we do that?

Thanks for your help

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

mgeorge — Wed, 24 Apr 2019 19:12:10 GMT

Hi Axelle Ribault‌,

In 4.11 we don't yet have support for passing in user-specified WebSocket connections, but you can use GeoEvent server to create a Stream Service.

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

Raul_Jimenez — Fri, 17 May 2019 10:19:34 GMT

Hi Axelle Ribault‌ it is possible to create a custom Stream Service, you can download the one we mentioned before.

Cheers

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

Raul_Jimenez — Fri, 17 May 2019 10:22:32 GMT

Finally Jorge Barrachina Gutiérrez fixed the issue we had, you can see how on this commit.

So feel free to download the code: https://github.com/esri-es/arcgis_websocket_server

FYI: We are still working on the "develop" branch.

Thanks Jorge, you rules!

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

AxelleRibault — Fri, 17 May 2019 11:36:18 GMT

Impressive, thank you for your work.

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

Raul_Jimenez — Fri, 09 Oct 2020 07:40:38 GMT

Finally you made it!!!, "webSocketUrl" is finally here! ^_^ -> StreamLayer | ArcGIS API for JavaScript 4.17

I have just tested it using your repo and it works , thanks!

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

mgeorge — Fri, 09 Oct 2020 17:37:14 GMT

Yes ... it took us a while but we were finally able get this in! Let me know if you run into any issues.

Re: Custom WebSocket Server + StreamLayer + JS API >= 4.9

Raul_Jimenez — Tue, 13 Oct 2020 12:03:35 GMT

Thanks Matt George !

I just noticed a weird blinking when the timeInfo was not set properly -> check this video (this is Chrome but It does the same on Firefox). I first used this code (fields and timeInfo were wrong).

After fixing it is worked as expected ^_^.

Thanks!

P.S: your code also worked