I've got a simple custom websocket server that adds a user to a specific websocket group during the handshake.
Using a constructor similar to the below:
new StreamLayer({
webSocketUrl: "wss://somewebsocket",
definitionExpression: "test=1",
...
});
even though my server is responding to the handshake with the correct filter, as seen below:

The stream layer is logging the below message:
[esri.layers.graphics.sources.connections.WebSocketConnection]
- t {name: 'websocket-connection', details: undefined, message: "Tried to set filter, but server doesn't support it"}
- details: undefined
- message: "Tried to set filter, but server doesn't support it"
- name: "websocket-connection"
- [[Prototype]]: C
I debugged the minified WebSocketConnection class and found that the code is doing something like: config.filter === response.filter.
javascript does object comparison by ref, so simply checking if the response object does not equal config object will never succeed.