Stream Service - WebSocket issue

[Nadiia_Matsiuk]

Hello everyone!

Our team already has some experience with GeoEvent Manager. Currently, we are facing a problem with using the stream service. When setting up the service, we refer to the tutorial 'Stream Services in GeoEvent Server Tutorial'.

I use the Send Features to a Stream Service output. It's helping us to publish the stream service correctly and successfully. There is nothing wrong with that.
The problem occurs when I try to subscribe to my service via REST endpoint (Subscribing to receive event data as an HTML client). When I click the 'subscribe' button, the text 'You have unsubscribed' is always returned.
Also, when the GeoEvent service is enabled, I see that objects are passed to the output. But I don't see any objects of this service (ArcGIS Pro, Map Viewer etc.).

We repeated this option in different browsers (Google Chrome, Mozzila Firefox, Microsoft Edge). The same error is displayed everywhere.

Here's the error text in the browser console:

'WebSocket connection to 'wss://<...>:6143/arcgis/ws/services/<...>/StreamServer/subscribe?token=<...>' failed: '

The 'netowrk' tab of the browser:

NS_ERROR_WEBSOCKET_CONNECTION_REFUESED
500 Server Error

We have read all the blog articles and tried to understand and find our own solution to configure the 'WebSocketContextURL' parameter on ArcGIS Server. But any of our changes do not work.

We also examined the GeoEvent Manager log. I will show you the sequence of responses to the place that probably shows the true error:

org.eclipse.jetty.server.session sessionHandler...
org.eclipse.jetty.servlet.ServletHandler servlet .... cxf-osgi-transport-servlet==org.apache.cxf.transport.servlet.CXFNonSpringServlet
org.ops4j.pax.web.service.jetty.internal.HttpServiceContext Handling request for ...
org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler handling request org.ops4j.pax.web.service.jetty.internal.HttpServiceRequestWrapper...
org.eclipse.jetty.servlet.ServletHandler chain=null
.esri.ges.rest.service.interceptor.AuthorizationInterceptor Authorization successful for ....
org.eclipse.jetty.io.ssl.SslConnection >fill SslConnection@5f080142::SocketChannelEndPoint
org.eclipse.jetty.io.ssl.SslConnection fill NOT_HANDSHAKING

So we see that there is no handshaking. But we cannot understand why.

We also refer to the support solution suggestions.

We installed an unsigned certificate on ArcGIS Server. This did not help.
We changed the Server Security Configuration to 'HTTP and HTTPS' (we cannot set the type to HTTP due to security requirements). This also did not change the situation.
The versions of our ArcGIS Server and Geoevent Server are 10.9.1.

We can also say that all this does not work in the AWS environment either.

Currently, we are using the data from the community article to test the stream service. We are doing this to make sure that the problem is not with our data.

Please, can anyone help? Any advice would be appreciated, for example:

• Is there any mandatory WebSocket configuration on ArcGIS Server, otherwise why it doesn't work in our case?
• In this case, how can I correctly diagnose ArcGIS Server/GeoEvent server to find the cause of the problem?
• Are there any topics that we should dismiss as the cause of the problem?
• In which direction should we move? ArcGIS Server, networking, etc.

I'll also attach a screenshot from my browser if it helps.

We are administrators of this system ourselves, so we have nowhere else to turn 🙂

Thank you for any help!

[GregoryChristakos]

Hi @Nadiia_Matsiuk - At a glance it looks like there's a certificate/trust problem here.

When you access GeoEvent Manager on the machine where GeoEvent Server is installed, does the browser present you with certificate errors? If not, can the same be said when accessing GeoEvent Manager from another machine entirely? I often found that a failure to subscribe to a Stream Service is 9 out of 10 times due to SSL/trust and your errors seem to indicate this. 

This is likely going to require a deeper dive into your environment to resolve, and so I would definitely recommend reaching out to our Support Services for assistance, but just off the cuff here, here are some things I would look into if I were in your shoes:

1. Ensure you have the latest patch installed for 10.9.1. This would be patch 4. We did some work on stream services as a part of this cumulative patch. None of the issues are directly related to what you're reporting, but this is a good start nonetheless.

2. If you can hit GeoEvent Manager and are getting SSL/certificate errors in the browser, I would look into addressing this. If you have a domain certificate you can work with, that is by far more preferable than using a self signed certificate. I often tell folks that a self signed certificate is analogous to writing your name on a piece of paper and trying to get through airport security with it as a method of identification. Its really only good for you, or to bring this back to GeoEvent Server, its only good for doing anything on the machine/browser where GeoEvent is installed and running. Forcing the browser to "trust" the self signed certificate isn't guaranteed to work either unfortunately.

- It'll be telling pretty quick if you can subscribe to the stream service from the browser where GeoEvent is installed/running, but cannot from anywhere else.

3. If you are using a domain level certificate, make sure you have the entire certificate chain imported into the ArcGIS Server admin endpoint. The root, any intermediates, and the cert itself. I've seen issues where just bringing in the cert alone isn't sufficient for establishing the chain of trust.

4. This might be a given, but did you try restarting GeoEvent Server following the certificate import you did on ArcGIS Server? Can you see if GeoEvent Server is presenting the new certificate when navigating to GeoEvent Manager?

These are a few things I would look into checking myself, but as I said before it might be best to reach out to Support Services where an analyst can take each of these suggestions further as needed. Hope this helps.

[Nadiia_Matsiuk]

Hi @GregoryChristakos!
Thank you for your advice!

Here are some of the results of our work:

1. We have installed the patch for 10.9.1. This did not solve our problem.

2. We do not get any errors regarding the certificate.
I will add a clarification: the machine with ArcGIS Server is running IIS with a web adapter. Maybe it's info provide some additional way to solve the issue.
ArcGIS Server is currently configured for http and https.
Web socket request currently looks like this: wss://<...>:6143/arcgis/ws/services/{service_name}/StreamServer/

3. Yes, ArcGIS Server was restarted after the certificate was imported. Again, this time there are no errors regarding the certificate.

Please tell me if you know of a way to solve this problem? Maybe we need to focus on ports (http/https, redirecting, url rewrite on IIS...)? 

Thank you!