EMedina-esristaff

Create a WSS Sever with Node.js and Express

Blog Post created by EMedina-esristaff Employee on May 1, 2019

Not too long ago, I was working on a GeoEvent Server issue that involved the Subscribe to an External WebSocket for JSON Input Connector. The problem was that a WSS server sent out heartbeat messages - these did a great job of keeping client connections alive but, as the heartbeats did not match any definitions, would also cause GeoEvent to log errors  The solution was to account for the heartbeat messages in the definitions and remove the associated field in the final output with a Field Reducer.

 

Anyhow, troubleshooting the issue led me to faithfully reproduce the problem by standing up my own WSS server. For that, I used Node.js and the ws package to create a server that would send some sample data to a client upon connection. This was great for quick testing, but it occurred to me that I might build upon the idea and turn it into a more user-friendly Web Application. So, I threw the Express web framework into the mix and came up with the attached. If you're also using GeoEvent Server, you might find this useful for testing/evaluating with following Input Connectors:

  • Subscribe to an External WebSocket for JSON
  • Subscribe to an External WebSocket for GeoJSON
    • To work with this connector, you'll need to make some adjustments to the file upload option.

 

Notes:

  • The application consists of an HTTP(S) server and a WS(S) server
  • You can either type your JSON or upload a .json file
  • You can either stream the JSON continuously (it'll just cycle through the events) or send it once

 

Part I: Basic Preparation

Start by downloading/installing Node.js if you don't already have it: https://nodejs.org/en/download/

 

Once installed, in a terminal/command prompt execute the following to install express-generator. This will make things easier by creating an application template.

npm install express-generator -g

 

From there,  execute the following to generate a template that uses pug/jade views. You'll run this where you want the application to reside:

express <name of your app> --view=pug

 

If successful, you'll be prompted to change directories. Do so, but before running npm install (as directed in the prompt) make a quick update to the package.json file.

 

Update package.json dependencies to include multer and ws:

"dependencies": {
    "cookie-parser": "~1.4.3",
    "debug": "~2.6.9",
    "express": "~4.16.0",
    "http-errors": "~1.6.2",
    "morgan": "~1.9.0",
    "multer": "^1.4.1",
    "pug": "2.0.0-beta11",
    "ws": "^6.2.1"
}

  • multer is used allow for upload of .json files
  • ws is used to create the WS/WSS server.

 

With that done, in the application directory run the following to install the application dependencies:

npm install

 

 

Part II: Creating the Application

Shown below are the the files in the application directory that will need to be updated/created (this is already done in the attached zip):

 

 

I'll briefly go over the updates in sections:

 

Application Setup:

app.js

The express template creates routes for index and users. You'll get rid of users and create new routes for stream-text and stream-file (you can call these whatever you want). One will be used for typed text, the other for file uploads.

Update the below sections:

var indexRouter = require('./routes/index');
var strtxtRouter = require('./routes/stream-text');
var strfileRouter = require('./routes/stream-file');
app.use('/', indexRouter);
app.use('/stream-text', strtxtRouter);
app.use('/stream-file', strfileRouter);

 

bin/www

This is essentially the piece that lifts things off the ground - the application web server. There's not a whole lot you need to do here unless you want to change the port used to run the application or configure SSL

 

To change the port, update this line:

var port = normalizePort(process.env.PORT || '3000');

 

By default, the application is set to use HTTP. To use HTTPS, you'll need a valid certificate. Simply comment out the HTTP lines and use these lines instead:

var https = require('https');
var server = https.createServer({
  cert: fs.readFileSync('/path/to/cert.crt'),
  key: fs.readFileSync('/path/to/key.key')
  }, app);
  • The above example uses a certificate and key - if you have a PFX or some other format just take a quick look online to see how to use it with Node.js.

 

As a side note, you could define the WS/WSS Server here - I chose not to do this as I already had the WSS server .js file created and it was easier (and cleaner) to plug in the file.

 

 

WS(S) Server and Uploads:

support/broadcast.js

This defines the WS(S) server and is written such that you could just use this file by itself to create the server. In that scenario, you would just include a binding for your JSON data and provide that to the relevant function. Again, you may configure SSL to create a WSS server or leave it as a WS server:

const WebSocket = require('ws');
const http = require('http');
const fs = require('fs');
const server = http.createServer();
// FOR SSL:
// const https = require('https');
// const server = https.createServer({
//   cert: fs.readFileSync('/path/to/cert.crt'),
//   key: fs.readFileSync('/path/to/key.key')
// });

const wss = new WebSocket.Server({ server });
wss.on('connection', (ws) => {
     console.log('Client connected');
     ws.on('close', () => console.log('Client disconnected'));
});

var interval = null;

let send_to_clients = data => {
  wss.clients.forEach((client) => {
    data.forEach(function(entry){
      console.log("entry ",entry);
      client.send(JSON.stringify(entry));
    }) 
  });
}

let prepare_data = json_in => {
  let data = JSON.parse(json_in);
  if (data.constructor !== Array) data = [data];
  return data;
}

let send_data = json_in => {
  let data = prepare_data(json_in);
  send_to_clients(data);
}

let stream_data = json_in => {
  let data = prepare_data(json_in);
  interval = setInterval(() => {
    send_to_clients(data);
  }, 1000);
}

let clear_stream = () => {
     clearInterval(interval);
}

server.listen(3001);
module.exports = {
  wss: wss,
  stream_data: stream_data,
  send_data: send_data,
  clear_stream: clear_stream,
  interval: interval
}

Here the WS(S) server listens on port 3001 and does the following:

  • Prints a message when a client connects to the server.
  • Sends defined data to all connected clients - it will send each event separately.
    • It'll use either send_data to send defined data once or stream_data to cycle through the data and send it continuously (here it is set to 1000ms but that can be modified).
  • clear_stream is used to stop any stream that might be running.

 

support/upload.js

Using the multer package, this script provides for file uploads in case you want to use a .json file. Uploads are set to go to public/uploads. If you wish to change the location, create the directory and update line 5 accordingly.

const multer = require('multer');

var storage = multer.diskStorage({
    destination: (req, file, cb) => {
      cb(null, 'public/uploads')
    },
    filename: (req, file, cb) => {
      cb(null, file.originalname)
    }
});

var upload = multer({storage: storage});

module.exports = upload;

 

 

Application Routes:

routes/index.js

This is the application's main route that processes submitted JSON and sends/streams it from the WS(S) server.

const express = require('express');
const router = express.Router();
const fs = require('fs');
const wss = require('../support/broadcast');
const upload = require('../support/upload');
const stream_data = wss.stream_data;
const send_data = wss.send_data;
const stream_file = wss.stream_file;
const clear_stream = wss.clear_stream;
var interval = wss.interval;

router.get('/', function(req, res, next) {
  clear_stream(interval);
  res.render('index', { title: 'WSS Server' });
});

router.post('/stream-text', (req, res) => {
     try {
    if (req.body.send==='send_once'){
        send_data(req.body.json_in);
        res.render('stream-text', {
          title: 'Send Status',
          status: 'Successful - event(s) sent once.' });
    } else {
        stream_data(req.body.json_in);
        res.render('stream-text', {
          title: 'Stream Status',
          status: 'Successful - streaming...' });
    }
  } catch (e) {
      res.render('stream-text', {
        title: 'Status',
        status: `${e.name} : ${e.message}` });
  }
});

router.post('/stream-file', upload.single('file-to-upload'), (req, res) => {
     let raw_data = fs.readFileSync(req.file.path,'utf8');
     try {
    if (req.body.send==='send_once'){
        send_data(raw_data);
        res.render('stream-file', {
          title: 'Send Status',
          status: 'Successful - event(s) sent once.' });
    } else {
        stream_data(raw_data);
        res.render('stream-file', {
          title: 'Stream Status',
          status: 'Successful - streaming...' });
    }
     } catch (e) {
      res.render('stream-file', {
        title: 'Status',
        status: `${e.name} : ${e.message}` });
     }
});

module.exports = router;
  • /stream-text is the route used for text.
  • /stream-file is the route used for uploads.
    • If an option to send once is ticked, then the send_data function will be used.
  • If there are no issues with input JSON then a success message is displayed. If there are issues, then the captured exception will display. Most of the exceptions will probably be syntax errors so you could be more specific and use e instanceof SyntaxError in the try/catch.

 

 

routes/stream-file.js & routes/stream-text.js

These routes are basically the same so update them as you see fit:

var express = require('express');
var router = express.Router();

router.get('/stream-file', function(req, res, next) {
  res.render('stream-file', { title: 'WSS Server' });
});

router.get('/', function(req, res, next) {
  res.render('index', { title: 'WSS Server' });
});

module.exports = router;

Not a lot happens here besides rendering the stream-file/stream-text views and rendering the index if you choose to stop the stream/return to the main page.

 

 

Application Views:

public/stylesheets/style.css

Here you can set whatever styling you want. In my example I only added the below:

.cwidth {
  width: 30rem;
}

.sz {
  height: 4rem !important;
}

 

views/layout.pug

This is the pug template used for all other pages. Besides the style.css, Bootstrap is used to create the web pages.

doctype html
html
  head
    title= title
    link(rel='stylesheet', href='/stylesheets/style.css')
    link(rel='stylesheet', href='https://maxcdn.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css')
    script(src='https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js')
    script(src='https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js')
    script(src='https://maxcdn.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js')
  body
    block content

 

views/index.pug

This is the main page where users will submit their JSON to stream/send from the WS(S) server. Defined here are two forms, one for typed text and one for file uploads.

extends layout

block content
  h1= title
  p Listening on Port 3001
  div.top
  .card.cwidth
    .card-body
      h5.card-title Copy/paste JSON for Testing:
      form(method='POST' action='/stream-text')
        input(type='checkbox', name='send', value='send_once')
        |  Send Once
        br
        .input-group
          textarea.form-control(rows='3', placeholder='Paste JSON here', name='json_in')
          .input-group-btn
          button.btn.btn-secondary.btn(type='submit') Submit
  br
  .card.cwidth
    .card-body
      h5.card-title Upload JSON for Testing:
      form(method='post', action='/stream-file', enctype='multipart/form-data')
        input(type='checkbox', name='send', value='send_once')
        |  Send Once
        br
        .input-group
          .card
            input.btn(type='file', name='file-to-upload', required, accept='application/json')
            .input-group-btn
          button.btn.btn-secondary.btn-sm(type='submit') Upload
  div.footer

 

The resulting page looks like this:

 

views/stream-file.pug & views/stream-text.pug

As with the routes these views are essentially the same - here is reported the status and button to stop the stream/return to the index page is provided:

extends layout

block content
  h1= title
  div.top
  .card.cwidth
    .card-body
      form(method='GET' action='/')
        .input-group
          textarea.sz.form-control(rows='3')=status
          .input-group-btn
          button.btn.btn-secondary.btn(type='submit') Stop/Return
  div.footer

 

The resulting pages look like this:

 

 

Part III: Using the Application

To run the application, in the application directory (where app.js is) issue this command:

npm start app.js

  • Use npm start in development.

 

If you have no errors in your code you should see something like this:

  • Visit the index page and you'll see the corresponding GET requests logged.

 

If you intend to use the WS(S) server for GeoEvent Server testing, go to the GeoEvent Manager and create a Subscribe to an External Websocket for JSON Input Connector. Enter the information as below:

 

  • Please note only basic settings are shown. If you plan to construct geometry you can do so by setting the advanced options.

 

Upon creating and starting the Input Connector, you should see the "Client connected" message in the Node.js console. Submit some JSON and you'll see something like this:

 

 

Your input counts in GeoEvent Manager should be increasing as well - that's it! Hopefully you found this interesting and can build upon some of the ideas presented here.

Attachments

Outcomes