Geoevent WebSockets in 10.6 with ARR

[SebastianCabrera]

I have a problem with WebSockets being proxied with ARR and URL rewrite in Geoevent Server 10.6, I cannot subscribe to the stream services using Chrome, Firefox, but it works on Edge and IE. I had it working fine in 10.4.1 and 10.5.1, configured as described in https://community.esri.com/community/gis/enterprise-gis/geoevent/blog/2016/02/05/using-iis-applicati.... I upgraded to 10.6 and stopped working. My first guess was any certificate issue, but requests are being redirected to geoevents 6180 port. Any change on how WebSockets are being handled in 10.6, RJ Sunderman?

I had to downgrade to 10.5.1 and subscrption works fine.

[RJSunderman]

There were two changes made to stream services for the 10.6 release:

1) The websocket library was upgraded to jetty 9.x.x for the 10.6 ArcGIS release. This could impact external clients, but when we tested with web maps on Chrome we didn’t see a problem.

2) The way the stream service outbound transport sends events to the websocket server changed in the ArcGIS 10.6 release. Basically, if the client appears to be making a request to the local ArcGIS Server, we call a method to send data directly to the websocket server rather than sending it through the internal websocket.

This second change should be internal, and should not affect how clients connect. Unless IIS/ARR is a contributing factor.

You indicate the problem is only seen with Chrome and Firefox. That would usually suggest a certificate issue or a JavaScript client issue, since the issue is not observed on other browsers.

Nikhil Shampur encountered something similar recently using stream services with IIS/ARR in Esri's cloud deployments. He told me that they had to change their ARR proxy rules to route WSS and HTTPS requests to the HTTP endpoint (6180) of GeoEvent. Client requests made via the load balancer must be made via HTTPS/WSS and through IIS/ARR to GeoEvent over HTTP.

Please consider opening an incident with Esri Tech Support to help track this issue.

- RJ

[SebastianCabrera]

Thanks for your quick response rsunderman-esristaff

I´ll consider opening an incident with Tech Support. For your information, I kept doing some tests, and discarded certificates issues, and I also tried using only http requests to the stream and setting the WebSocketContextURL to ws://fqdn and still doesnt work in Chrome (subscribes and inmediately unsubscribes). 

I set up a reverse proxy using nginx, and subscription works in Chrome. So I came to the conclusion that there must be some incompatibility with the combination of 10.6, IIS and Chrome. I´ll keep investigating.

Thank you

[DavidStajan]

I am also facing a similar issue with my Azure Vm hosting 10.6.  I applied all the same rules and configuration to IIS/ARR and AGS as previous environments that worked for 10.5.1 but can't get the streamservice to subscribe with 10.6.  Would you be able to elaborate on the rules that Nikhil applied in their environment to get things working?

Thanks,

David

[BruceDodson · Accepted solution]

I think I have figured out the cause of this. It has to do with way IIS + ARR handles the request, but does not appear to be related to the special path used in 10.6 for internal requests.

The issue seems to be with the Sec-WebSocket-Extensions header, for which Chrome and Firefox pass in a value of permessage-deflate. IIS + ARR passes through this header to the back-end, and the upgraded version of Jetty appears to handle this extension, but it appears IIS + ARR does not know what to do with it.

So, I added a rule to look for all requests with the Upgrade: websocket header, and strip out any Sec-WebSocket-Extensions header (actually, replace with a ";", which has the same effect) and it was then able to connect.

A few notes:

• The rule needs to come before your actual rewrite rules that redirect to the proxy. It can have Action: None and should not Stop processing other rules (or else it will never get to your actual rewrite rule).
• It is not necessary to add HTTP_SEC_WEBSOCKET_EXTENSIONS to the Allowed Server Variables to make this work, as long as the rule is defined within the Global Rules set (i.e. together with the ARR rewrite rules). You can if you want, but you don't need to.
• The rule could be restricted to specific URLs, e.g. if there are other WebSockets on the same server that are not proxied to GeoEvent and do not require this treatment. In my case I allowed it for all requests that contain the Upgrade request header and a non-blank Sec-WebSocket-Extensions request header.
• If the Sec-WebSocket-Extensions header included any other extensions that we wanted to pass through, they would also be lost. This could be prevented by looking for a Sec-WebSocket-Extensions header that contains the permessage-deflate extension specifically, and retaining everything except permessage-deflate. This is not hard to do via regular expressions and back-references. But then what if some other extension gets requested by browsers in the future that IIS + ARR also does not support? I figure it's better to strip out all extensions, for all WebSocket requests that get proxied by IIS, for maximum compatibility.
• The header could also have been rewritten within the same rewrite rule that forwards to the server farm; it didn't need to be a separate rule. I just found it easier and cleaner for our environment to define it that way.

Since this works, I can only assume that NGINX does not pass through this header at all, or else knows how to deal with per-message-deflated content, which might be why the issue was not detected in Esri's in-house testing, which I understand is primarily with NGINX.

Screen shot attached...

[BruceDodson]

(Updated screenshot to show the variable names in the Conditions and Server Variables sections, and fine tuned regular expressions in Conditions section.)

[JoëlHempenius3]

Thanks for this excellent answer!

After implementing your rule I got another error message in chrome console: Error during WebSocket handshake: Incorrect 'Sec-WebSocket-Accept' header value

Turns out I had to install the Web sockets role for IIS as well in Windows to enable Websockets for ARR. If somebody else also has this error, you might want to look at the Server Roles -> Web Server (IIS) -> Web Server -> Application Development -> WebSocket Protocol (Windows Server 2012R2 in this case)