How to deploy multiple Node.js Socket.IO servers with Nginx and SSL

A late post this time. I spent a good part of the past week figuring out how to deploy one or more Socket.IO-based Node.js servers using Nginx. Since it was about deployment, SSL was an important factor too. I write this post because of the sheer amount of Googling and trial-and-error I had to go through before I finally had a solution working. The primary problems in finding a straightforward solution via search engines were-

1. SSL – Especially since my application requires re-direction of POST requests as they are; something a simple rewrite command in Nginx wouldn’t accomplish.

2. Socket servers – Deploying a simple Node.js server is one thing, deploying a socket-based server is something different altogether.

3. Multiple Socket-servers – This was undoubtedly the trickiest part. To re-direct the incoming request to the appropriate server, and present it to the server in a format it understands, was the most difficult job (atleast for me). I experimented with a lot of Nginx rewriting, Node.js namespacing, etc., before I finally found an answer that worked.

So, here goes the procedure involved….

Step 1. Write your Node.js code(obviously)

For the sake of this tutorial, I will use two simple echo servers. Each is programmed to listen to its own port.

Server-1 (node1.js)


//PORT to connect to
const PORT = 3001;

//Instantiate socket server
var app = require('http').createServer().listen(PORT);
var io = require('socket.io').listen(app);

//Simple echo server
io.on('connection', function(socketconnection){
	socketconnection.send("Connected to Server-1");
	
	socketconnection.on('message', function(message){
		socketconnection.send(message);
	});
});

Server-2 (node2.js)


//PORT to connect to
const PORT = 3002;

//Instantiate socket server
var app = require('http').createServer().listen(PORT);
var io = require('socket.io').listen(app);

//Simple echo server
io.on('connection', function(socketconnection){
	socketconnection.send("Connected to Server-2");
	
	socketconnection.on('message', function(message){
		socketconnection.send(message);
	});
});


Step 2. Install PM2 for running Node.js servers

There are many options for this; I use PM2 for its simple commands and nice-looking interface. PM2 also helps ‘watch’ the Node.js servers you deploy, so that they can be restarted in case of failure or code changes. It also offers many other options for the way you want your server to function, but I won’t go into those details here.

Install PM2 using

npm install pm2 -g

Once installed, you can start your servers(with watching enabled) as follows:

pm2 start --watch node1.js
pm2 start --watch node2.js

Depending on your setup, you might have to use sudo with the above commands. Click here if you want to know more PM2 commands.

Step 3. Install and start Nginx

Pretty straight-forward.

sudo apt-get install nginx
sudo service nginx start

Step 4. Get your SSL certificates

You can either use un-verified self-signed certificates (good for development/testing), or buy ones from someplace like Comodo (essential for deployment).

To generate self-signed ones, do

sudo mkdir /etc/nginx/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt

This will put your certificates in /etc/nginx/ssl.

Step 5. Configure Nginx

Make a file called server_nginx.conf and put the following code in it-


#Upstream Node Server-1
upstream node1 {
	server 127.0.0.1:3001;
}

#Upstream Node Server-2
upstream node2 {
	server 127.0.0.1:3002;
}

#To redirect all HTTP traffic(keeping requests like POST intact)
#to HTTPS
server {
	listen 80;
	server_name localhost;

    location / {
  		return 307 https://localhost$request_uri;
	}
}


#The actual HTTPS server
server {
	listen 443;
    ssl on;
	server_name localhost;

	#SSL certificates
    ssl_certificate /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /etc/nginx/ssl/nginx.key;

	#For Server-1
	location /server1/ {
		#Configure proxy to pass data to upstream node1
		proxy_pass http://node1/socket.io/;
		#HTTP version 1.1 is needed for sockets
    	proxy_http_version 1.1;
    	proxy_set_header Upgrade $http_upgrade;
    	proxy_set_header Connection "upgrade";
	}

	#For Server-2
	location /server2/ {
		#Configure proxy to pass data to upstream node2
		proxy_pass http://node2/socket.io/;
		#HTTP version 1.1 is needed for sockets
    	proxy_http_version 1.1;
    	proxy_set_header Upgrade $http_upgrade;
    	proxy_set_header Connection "upgrade";
	}

}

You will now have to make Nginx aware of this configuration by symlinking this file into /etc/nginx/sites-enabled. To do that, do

sudo ln -s /path/to/server_nginx.conf /etc/nginx/sites-enabled/

Then restart Nginx

sudo service nginx restart

Thats about it. But before we go to the client code, a few points to note about the above Nginx .conf file:

1. You can use rewrite on line 18; I do it this way because it preserves non-GET requests as they are. If you know a better way to go about it, do let me know :-).

2. Defining the Node.js servers in upstream blocks is good practice, especially since it helps do easy load-balancing between identical Node.js servers in the future. More on that here.

3. If you buy SSL certificates from a vendor, you can modify lines 30 and 31 as necessary.

4. Observe lines 36 and 46. The trailing /socket.io/ (including the trailing backslash) is essential, since it replaces the original URI and expresses the request in a format that Socket.IO will understand.

5. Depending on your application, you might have to add more code to the .conf file. For example, if your application involves long intervals where no reading takes place on the socket connection, you might have to increase the proxy read-timeout period.

6. Ofcourse, during actual deployment, you will have to replace locahost with your actual domain name.

Finally, heres a client code to connect to a server and send a message:


var io = require('socket.io-client');
var conn = io.connect('https://localhost', {path: '/server1'});

conn.emit('message', 'Some message');
conn.on('message', function (data){
	console.log(data);
});

Note the ‘path’ parameter thats conveyed during connection, and the explicit use of https in the address.

Thats it for this time :-D. Hope this helps someone, and saves him/her the trouble of searching the internet for a complete day trying to accomplish (what is really) a trivial thing. Thanks for reading!

Simple production-time debugging in Django – and better error handling

We all know how amazingly exhaustive Django debugging can be, when you enable the Debug = True option in your settings file. It gives you a full traceback, complete with the str-ed versions of local variables. However, this post deals with the production-time scenario, when you might encounter an un-anticipated error in your code. Such errors can arise out of various reasons, such as-

1. A kind of input you did not anticipate.

2. A logical error of some kind, usually occuring out of boundary cases (like a sqrt function coming across a negative value)

3. (Worst kind) Some type of low-probability scenario you forgot to test.

In such cases, Django serves the 500 status code along with a “Server Error” message. All this is good, and if your application doesnt have any state variables that may get screwed, or low-probability errors aren’t that important to you, then you can simply ignore the rest of this post. However, if you would want atleast a minimalistic note being made on the server when something goes wrong, then you can set up a simple debug-logging system as a part of your Django project.

Heres how you go about it-

Step 1. Create a folder called debug in your Django project’s main directory (the one that contains the manage.py file).

Step 2. Make an empty __init__.py file inside the new folder, which will tell Python thats it contains relevant code.

Step 3. Make a models.py file in the folder, and add the following lines to it:


from django.db import models


class ExceptionLog(models.Model):
    """
    Models any error occuring on the server.
    """
    timestamp = models.DateTimeField('Time Stamp')
    view = models.CharField('View', max_length=30)
    exceptionclass = models.CharField('Exception Class', max_length=60)
    message = models.CharField('Exception Message', max_length=100)

This sets up the basic database model for logging of Python Exceptions. Modify it as you wish, especially the names, max lengths and stuff.

Step 4. Make a file named admin.py in the folder, and add the following code to it:


from django.contrib import admin
from debug.models import ExceptionLog


class ExceptionLogAdmin(admin.ModelAdmin):
    list_display = ('timestamp', 'view', 'exceptionclass',
                    'message')
    list_filter = ('view', 'timestamp')
    search_fields = ['message', 'exceptionclass', 'view']

admin.site.register(ExceptionLog, ExceptionLogAdmin)

The attributes are again to your taste – If you know the basics behind configuring the Django admin page, you already know what you want and how to get there. If you don’t, reading up on the Django tutorial will help.

Step 5. This is where we add the code that makes things work. Add a file called decorators.py in the directory, and add the given code to it:


from debug.models import ExceptionLog
from django.utils import timezone
from django.http import HttpResponse


def log_exceptions(view_name):
    """
    Logs all the exceptions occuring in a Django view, to the
    ExceptionLog model.
    'view_name' denotes an identifier for the view that is
    being debug-logged.
    """

    def real_decorator(actual_view):
        """
        This is the actual decorator.
        """

        def wrapped_view(request):
            """
            This is the version of the view that will monitor
            itself for any un-expected Exception, and maintain basic
            logs of the same.
            """
            try:
                #Run the view code
                response = actual_view(request)
                #If a response is generated without any Exception
                #coming up, return it
                return response
            except Exception as e:
                #If an unexpected Exception occurs, make a debug entry
                #and save it
                debug_entry = ExceptionLog(
                    timestamp=timezone.now(),
                    view=view_name,
                    exceptionclass=str(e.__class__),
                    message=str(e))
                debug_entry.save()
                #Return the Server Error(500) status code
                return HttpResponse(status=500)

        return wrapped_view

    return real_decorator

This code uses the Python ‘magic’ called decorators with arguments. Frankly, I had never implemented Python decorators (used yes, during SymPy work), let alone with arguments, before this. But trust me- unless you have a very special case, this is the way to go for what we want to achieve here.

The decorator basically takes in as argument the name of the view, so that its known during logging. It encapsulates the working of the actual Django view thats being decorated in a try...except block, and logs the time-stamp, view name and the most important Exception details, in the event that one is raised.

Step 6. Add 'debug' to the 'INSTALLED_APPS' tuple in your settings.py Django file. This officially recognizes your new debug-logging app as a part of your complete server code.

Then, before re-starting the server, dont forget to do syncdb or migrate (depending on which Django version you use) on your project. This will essentially register the new models on the server, and make the details appear on your Django admin page.

Step 7
. To set up any view in your project code for the debug-logging, just modify it as follows-

...other imports...
from debug.decorators import log_exceptions

@log_exceptions('Some View')
def some_view(request):
    ...view code...

If you also use the @csrf_exempt decorator on your view(s), make sure the log_exceptions decorator lies below it.

Voila! You are done! Now, whenever your decorated views raise an Exception, the relevant information will be logged on your Django admin page.

Ofcourse, this does not replace what real debugging does, nor is it as exhaustive (no traceback provided, for starters). It just stores the very basic info about what went wrong, when and where(in terms of which view). However, it will not give you complete details like file name, line no. etc. In a production environment, debugging to that extent is stupid and (should be) unnecessary. That brings me to the second important point-

Whenever an Exception is raised in your Python code, there are two things you can do about it-

1. ‘Duck it’ – Not handle it, letting it propagate down the function stack, OR

2. ‘Handle it’ – Using something like a try...except block, either repair the damage caused by the error (say returning a default value), or raise another Exception that better describes what went wrong (most often, in ‘high level’ terms).

I implore you to do the latter in your code, atleast as much as possible. Why? It just makes your life way easier. In simple programming terms, the lesser the depth of your function stack when an Exception is raised, the easier it is for you to understand what went wrong. But ofcourse, if you already knew things were going to go wrong, you would not need this debugging tool :-P.

But even then, try not to let Exceptions come out from places like the NumPy core, where it becomes frustratingly difficult to know how things screwed up. It is always a good practice to raise good-quality Exceptions yourself, with custom error-messages that say what went wrong. As long as you can understand the problem from looking at the Exception class and message, you are good to go!

EDIT 1: Ofcourse you could use better frameworks meant for this like Sentry and Opbeat. I wrote this post just to show a way to do it quick-and-dirty, in your code itself. If needed, not re-inventing the wheel is ofcourse always better :-).

Efficient computation and storage of basic data statistics using Redis

This post describes a script for efficient computation and storage of the mean and variance corresponding to data from multiple sources. It uses Redis as a backend storage system. Though its written in Python(mainly cause I work with Django a lot), it can be translated into any popular language out there.

The basic assumptions are as follows:

1. You have data arriving from multiple sources, each possessing a unique ID(String) of some sort. For example, the location in case of weather data.

2. You need to store the mean and variance corresponding to non-overlapping time periods, like days(as given in the script).

3. You don’t want to/need to store the individual data points, just the averages. Moreover, you don’t want to access the permanent data storage (like a file on disk) everytime a new data point comes in, but only when the average stats need to be stored. The primary reason for this could be efficient resource usage.

4. You don’t want to store the process parameters as variables in the program, but rather using a better option like Redis. (Though you can tweak the script to do that too.)

Heres the script:


#Imports
from redis import StrictRedis
import datetime

#Redis Client for communicating with Redis
redisdb = StrictRedis()


def timestamp():
    """
    Returns a String that denotes a unique time-period.
    The output of this function determines the period over which
    the averaging is done.
    """
    return datetime.datetime.now().strftime("%d-%m-%Y")


def _fetch_state(identifier):
    """
    Fetches all the state data from Redis corresponding to the
    identifier string.
    If any part of it is not present, it returns empty data.
    Returns last date timestamp, the sum, the sum of squares,
    and the counter value (in that order).
    """

    #Get the string data from Redis
    last_timestamp = redisdb.get(identifier + '_last_timestamp')
    sigma = redisdb.get(identifier + '_sigma')
    squares_sigma = redisdb.get(
    identifier + '_squares_sigma')
    counter = redisdb.get(identifier + '_counter')

    #Check if any of the above is None(not present)
    #If not, parse the numbers
    if None in [last_timestamp, sigma, squares_sigma,
           counter]:
        #If any one is not available, the others are useless
        #Just reset the values
        last_timestamp = ''
        sigma = 0
        squares_sigma = 0
        counter = 0
    else:
        sigma = float(sigma)
        squares_sigma = float(squares_sigma)
        counter = int(counter)

    return last_timestamp, sigma, squares_sigma, counter


def _store_state(identifier, last_timestamp, sigma, squares_sigma,
    counter):
    """
    Stores the state data corresponding to the identifier, to Redis.
    """

    redisdb.set(identifier + '_last_timestamp', last_timestamp)
    redisdb.set(identifier + '_sigma', sigma)
    redisdb.set(identifier + '_squares_sigma',
        squares_sigma)
    redisdb.set(identifier + '_counter', counter)


def _permanent_store(identifier, mean, variance):
    """
    Stores statistical data to some kind of permanent storage
    (A file in this case).
    """
    f = open(identifier + '.txt', 'a')
    f.write(str(identifier) + ', ' + `mean` + ', ' + `variance` + '\n')
    f.close()


def record(identifier, value):
    """
    Records a numeric value corresponding to the identifier.
    """

    #Fetch state data
    last_timestamp, sigma, squares_sigma, counter = (
        _fetch_state(identifier))

    #Compute current timestamp
    current_timestamp = timestamp()

    if last_timestamp != current_timestamp:
        #Either a new time period has started, or
        #this is a new identifier, or
        #previous state data was lost for some reason.
        if counter != 0:
            #A new time period has started,
            #so compute parameters for previous one and store.
            counter = float(counter)
            mean = sigma/counter
            variance = squares_sigma/counter - mean**2

            _permanent_store(identifier, mean, variance)

        #Intialize state based on newly received data
        last_timestamp = current_timestamp
        sigma = value
        squares_sigma = value**2
        counter = 1

    else:
        #Same time period as before, update state
        sigma += value
        squares_sigma += value**2
        counter += 1

    #Store state
    _store_state(identifier, last_timestamp, sigma, squares_sigma,
        counter)

Some things to note:

1. The script is resistant to restarts (keeping in mind that any data that comes in during the down-time is lost). It’s also resistant to faults on Redis’ behalf, though any state data stored would then be lost. In both cases, the averaging information may be inaccurate, but not unavailable.

2. You can group sources together if you want (like averaging weather data from all sources in a region). You would just have to implement functionality to map each source ID to the relevant group ID.

3. The script computes averages over days, but using little creativity with the timestamp function, you can compute statistics over any custom time periods. In fact, you can modify the source IDs to tell the system what time interval to average over.

4. Though the input in the above code is single numbers, you can work with multidimensional data(using NumPy arrays/Pandas) and even other types of information extraction(that can work with streams without requiring historical data).

5. The script stores the data into a text file; you can obviously make it store the data to any other destination.

6. The code is pretty basic. To actually use it as a part of a bigger framework, I would suggest taking measures such as bundling the code into a class, implementing synchronization locks over the state data, etc. All that is upto you and your application ofcourse.

Thanks for reading!

Redis + Node.js + Socket.IO – Event-driven, subscription-based broadcasting

Recently, I have been working on building a server for broadcasting messages over socket connections. The basic design I had in mind was something like this-

1. The broadcast channels would basically be Redis Pub/Sub channels, with input coming from outside of the server(either a script or another server).
2. Each socket client would define the channels it wanted to listen to, via subscribe and unsubscribe requests to the server.
3. I did not want to instantiate a new Redis client per socket connection, but rather one Redis client per channel. This way, it could be shared(for listening) among all the socket connections subscribing to it.

Given below is the code for this server. The dependencies you will have to satisfy for running it are-
1. Redis
2. Node.js
3. The following Node.js modules- socket.io, redis.
If you don’t, some Googling will get it done soon enough 😛


//Port config
const PORT = 3000;

//Requires and main server objects
var redis = require('redis');
var socketio = require('socket.io');
var app = require('http').createServer().listen(PORT);
var io = socketio.listen(app);

//This object will contain all the channels being listened to.
var global_channels = {};

//Server Logic goes here
io.on('connection', function(socketconnection){

//All the channels this connection subscribes to
socketconnection.connected_channels = {}

//Subscribe request from client
socketconnection.on('subscribe', function(channel_name){
//Set up Redis Channel
if (global_channels.hasOwnProperty(channel_name)){
//If channel is already present, make this socket connection one of its listeners
global_channels[channel_name].listeners[socketconnection.id] = socketconnection;
}
else{
//Else, initialize new Redis Client as a channel and make it subscribe to channel_name
global_channels[channel_name] = redis.createClient();
global_channels[channel_name].subscribe(channel_name);
global_channels[channel_name].listeners = {};
//Add this connection to the listeners
global_channels[channel_name].listeners[socketconnection.id] = socketconnection;
//Tell this new Redis client to send published messages to all of its listeners
global_channels[channel_name].on('message', function(channel, message){
Object.keys(global_channels[channel_name].listeners).forEach(function(key){
global_channels[channel_name].listeners[key].send(message);
});
});
}

socketconnection.connected_channels[channel_name] = global_channels[channel_name];

});

//Unsubscribe request from client
socketconnection.on('unsubscribe', function(channel_name){
if (socketconnection.connected_channels.hasOwnProperty(channel_name)){
//If this connection is indeed subscribing to channel_name
//Delete this connection from the Redis Channel's listeners
delete global_channels[channel_name].listeners[socketconnection.id];
//Delete channel from this connection's connected_channels
delete socketconnection.connected_channels[channel_name];
}
});

//Disconnect request from client
socketconnection.on('disconnect', function(){
//Remove this connection from listeners' lists of all channels it subscribes to
Object.keys(socketconnection.connected_channels).forEach(function(channel_name){
delete global_channels[channel_name].listeners[socketconnection.id];
});
});

});

Lines 20-43 define the server’s behavior when a new request is made by a connected socket, to subscribe to a certain broadcast channel. A broadcast channel is nothing but a Redis client that subscribes to a given pubsub channel, with the added property of a collection of listeners. Each of these Redis clients is configured to send every message it receives on the pubsub channel, to all of its listeners. The global object global_channels maintains a mapping of channel_name to Redis client instance, so that it can be shared among listeners. Each socket client also maintains its own collection of channels it subscribes to.

On unsubscribing (lines 45-54), the socket client is removed from the Redis client’s listeners, and the Redis client is removed from the socket client’s connected channels. On disconnecting, the socket is removed from the listeners of all Redis clients it subscribes to.

Now, heres a small Node.js client that will connect to the above server, subscribe to somechannel, and unsubscribe as soon as the first message is received. You will need the socket.io-client Node module for this.


var io = require('socket.io-client');
var serverUrl = 'http://localhost:3000';
var conn = io.connect(serverUrl);

conn.emit('subscribe', 'somechannel');
conn.on('message', function (message){
console.log(message);
conn.emit('unsubscribe', 'somechannel');
});

The server code in this post is pretty basic, and its only job is to showcase the functionality I intended to show. For production/serious applications, there are quite a few changes you would want to make, such as-

1. Token/user-password based authentication. This could also ensure that every client can subscribe/unsubscribe only to a given/alloted set of channels.

2. Other security measures, such as disconnecting a client thats not subscribed to any channel for a given timeout.

3. Deleting channels with no listeners.

This is my first time writing proper Javascript/Node.js code, so do let me know if I have made some mistake or there are any optimizations possible to the code above 🙂 .

EDIT 1:

You could also go with one-Redis-client-per-sever (instead of channel), it would just mean storing the relevant mappings and implementing the routing logic in the ‘message’ event for the Redis client. Would provide considerable gains in case of a high number of channels.