StreamServer error/question

Question

I have a GeoEvent service that I am writing the output to a StreamService.  The service was working fine earlier in the week but is now it seems to constantly be starting and stopping.   The service appears to be fine on the ArcGIS Server, it's just the output connector that is stopping and restarting.   The error log has messages like this:

closedOut 1009:Text message size > 10240 chars java.io.IOException: closedOut 1009:Text message size > 10240 chars

at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameConnection.sendMessage(WebSocketConnectionRFC6455.java:437 [116:org.eclipse.jetty.websocket:7.6.9.v20130131]

at com.esri.ges.datastore.agsconnection.StreamServiceClientImpl.sendMessage(StreamServiceClientImpl.java:42)[241:com.esri.ges.framework.datastore.agsconnection-datastore:10.3.0]

at com.esri.ges.transport.streamService.StreamServiceOutboundTransport.receive

(StreamServiceOutboundTransport.java:274)[274:com.esri.ges.framework.transport.stream-service-transport:10.3.0]

at com.esri.ges.manager.stream.internal.OutboundTransportProxy.receive(OutboundTransportProxy.java:40

[325:com.esri.ges.manager.internal-streammanager:10.3.0]

at com.esri.ges.manager.stream.internal.OutboundStreamImpl.receive(OutboundStreamImpl.java:184

[325:com.esri.ges.manager.internal-streammanager:10.3.0]

at com.esri.ges.adapter.OutboundAdapterBase.receive(OutboundAdapterBase.java:44

[253:com.esri.ges.framework.adapter.adapter-api:10.3.0]

at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.flushQueue(FeatureJsonOutboundAdapter.java:116

[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]

at com.esri.ges.adapter.json.FeatureJsonOutboundAdapter.run(FeatureJsonOutboundAdapter.java:91

[258:com.esri.ges.framework.adapter.feature-json-adapter:10.3.0]

at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]

SharonMeier · Answer

RJ,

Thank you for the information.  I tried adjusting the web socket setting as you suggested.  Still getting the error.  Another odd thing is that I am unable to subscribe to stream service.  When I click on the Subscribe button I get a message at the bottom of the window stating that I have unsubscribed.  I have been able to subscribe to other stream services on a  different server setup, but not on this one.

RJSunderman · Answer

Hello Sharon -

I have not encountered the message java.io.IOException: closedOut you observed. But I have seen messages referring to an event message's size being larger than 10240 characters, my Stream Service output entering an ERROR state, and eventually restarting.

Usually when messages constructed from my event data are too large I see messages like:

* Cannot send a message to the Stream Service
* connection is already closed due to connection error
* Text message too large > 10240 chars

The WebSocket protocol is the communication standard underlying Stream Services. The 10.3.0 / 10.3.1 release of the GeoEvent Manager exposes a couple of properties you can use to configure the buffer size allocated to hold data being transmitted via the web socket and the maximum allowable size of a text message.

* Use GeoEvent Manager to navigate to the Site > Settings > Configure Global Settings page
* Scroll or search to locate the Stream Service Settings:

Edit the 'Maximum web socket text message size (bytes)' and the 'WebSocket buffer size (bytes)' properties. Start by increasing the values by a factor of 10x (from 10240 to 102400). You will want to keep an eye on your system resources to verify that increasing this limit is not burdening the JVM (Java virtual machine) GeoEvent is using.

It is my understanding that specifying a 102400 byte setting will allocate up to 100k for web socket messages. I’m not sure how frequently the allocated memory is released. If you were to increase the allocation default 100x (from 10240 to 1024000) you could potentially be reserving a full megabyte of the JVM's RAM allocation just for Stream Services use.

You can always subscribe to the Stream Service's REST endpoint (e.g. https://your-server:6443/arcgis/rest/services/your-service-name/StreamServer/subscribe) to see the messages being broadcast by you Stream Service to clients. This might help you get an idea of what your expected message size should be, so you know how to set the message size and buffer size property values.

You should not need to restart the GeoEvent Extension after editing the properties illustrated above. But you will need to close and reestablish any client connections so that those connections are aware of the changes. If you have clients subscribed to the Stream Service, simply unsubscribe and resubscribe. If your GeoEvent Stream Service output is running, stop it and then restart it.

Hope this information helps -

RJ