How to Create REST API Using Node.js – Part 3

“We are all tasked to balance and optimize ourselves.”

― Mae Jamison

Introduction

This is part 3 of the “How to Create REST API Using Node.js” series. After initiating the API project and integrating MongoDB on it, we will now implement data caching using Redis.

Also, with more integrations, we will also refactor our code to be more configurable using dotenv. We installed it before, but have not implemented it yet.

Lastly, I will share the source code from GitHub at the end of the post.

Requirements

Here are the requirements you have to complete before proceeding to the integration part.

Redis

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams.

https://redis.io/

You can find the installation steps here.

RedisInsight

If you would like to install a GUI for Redis, you can download RedisInsight here. After installing, you can create a new Redis Database (don’t get confused with MongoDB Database) to access later.

First, add a new database by clicking Add Redis Database, then Add Database.

Second, fill the fields like the screenshot below, then click Add Redis Database.

If you encounter the error below, you must start your redis server via CLI: redis-server.

Third, click on the newly created redis-local to browse your redis data.

Finally, click on Browser, here is the place where you can browse the created redis keys.

Part 3 – Data Caching Using Redis and Also Some Refactoring Because It’s Getting Complicated

Prerequisites

START FRESH (IF YOU WANT TO)

If you’d like to start fresh to learn this topic, please clone from this repository in GitHub.

Install Redis Library

Open your terminal, then type this command: npm i redis. If it runs successfully, it should look like this:

Start Redis Server

You should have started the Redis server already when testing your Redis installation. Or, maybe you set it so long time ago you forgot how to start it. So, here’s how to do it: Start your terminal, run redis-server. Easy. If it starts successfully, it should look like this screenshot below:

---

Connecting to Redis

Create Redis Connector Module

Let’s make a new file named redisConnector.js. In this file, we will code everything related in Redis from initiating connection, operations, and disconnecting. See this snippet below:

// redisConnector.js

const redis = require('redis');

class RedisConnector {
  constructor() {}

  connect(host, port) {
    const client = redis.createClient(port, host);

    client
      .on('ready', () => {
        console.log('Redis client is ready.');
      })
      .on('error', (err) => {
        console.log(`Redis Client Error: ${err}`);
      });

    Object.assign(this, { client });
  }

  disconnect() {
    if (this.client) {
      console.log('Closing Redis client...');
      this.client.quit(() => {
        console.log('Successfully closed redis client.');
      });
    }
    else {
      console.log('Redis Client do not exist!');
    }
  }
}

module.exports = RedisConnector;

First, we import the Redis library with const redis = require('redis');.

Second, we create a class named RedisConnector, then export the module with module.exports = RedisConnector; at the bottom of the file.

Third, in the RedisConnector class, we create two functions: connect() and disconnect().

In connect(), we create a new Redis client with port and host arguments taken from connect() parameters. Then, we set at least two event listeners with .on() for the events ready and error. For now, we keep it simple just by logging “Redis client is ready.” and “Redis Client Error: <error_detail>” for the event callback function. Lastly, we assign client as the class property to enable client to be used in every functions in RedisConnector later on.

Then, on disconnect(), we simply call the client if it exists with this.client. To disconnect from the Redis server we just need to call this.client.quit(), with the callback function triggered when it’s completed.

Import Redis Connector

Then, we modify index.js to start a connection with RedisConnector, at the beginning of starting the API.

// index.js

...
const MongoDbConnector = require('./mongoDbConnector');
const RedisConnector = require('./redisConnector');

...

const collection = 'cil-users';

const redisConnector = new RedisConnector();
redisConnector.connect('localhost', 6379);

...

['SIGINT', 'SIGTERM'].forEach((signal) => {
  process.on(signal, async () => {
    ...
    mongoDbConnector.disconnect();
    redisConnector.disconnect();
    ...
  });
});

First, we import the newly created RedisConnector from redisConnector.js with const RedisConnector = require('./redisConnector');

Second, we create a new RedisConnector object called redisConnector with const redisConnector = new RedisConnector();

Then, we connect to the redis server with redisConnector.connect(). It requires the host and port arguments. For the host, we will connect to localhost, so put ‘localhost’ there. For the port, we use the default Redis port 6379.

Skip to the end of the file, where we set the mongoDbConnector.disconnect() before, add redisConnector.disconnect() to disconnect from the Redis server on stopping the API application.

Test the Code

If everything works correctly, you should see on your terminal like this screenshot below:

And here is what would happen if you forget to start the redis-server:

---

Integrate Cache On GET /user/:id

Frequently requested data should be stored in a cache, which enables it to be loaded faster.

Let’s say we have some particular users which data is often requested through the API. Sure, MongoDB can do the job well, but we want to do better. It would improve the user’s experience. And so, we shall modify our GET /user/:id route.

Redis Connector

Let’s code the two most important things first: get() and set() data on Redis. See this example below:

// redisConnector.js

class RedisConnector {
  ...

  async set(key, data, expiry) {
    return new Promise((resolve) => {
      this.client.set(key, JSON.stringify(data), 'EX', expiry, (err, reply) => {
        resolve(reply);
      });
    });
  }

  async get(key) {
    return new Promise((resolve) => {
      this.client.get(key, (error, result) => {
        if (result) {
          resolve(JSON.parse(result));
        }
        resolve(null);
      });
    });
  }

  ...
}

Both get() and set() belongs to RedisConnector class.

set() requires three parameters: key, data, expiry. key is the identifier for data, enabling the data to be accessed from get() later on. expiry is the time to live for the data in seconds.

The set() function is required to be asynchronous, because we want to make sure the data is set correctly before continuing the next operations. Therefore it returns a promise, in which a resolve callback will be executed by the callback of this.client.set().

In this.client.set(), the arguments are separated by key and value for each comma. For example, key‘s value is data which is stringified, and the key EX for setting expiry time, with expiry as the value. The last argument is a callback function to get the result of executing this.client.set().

Next, the get() function requires only a key parameter to find the data. It returns a promise which will call a resolve callback after executing the callback of this.client.get(). If the data exists, the data will be parsed as JSON. But if not, it will return as null.

index.js

We will continue to index.js. As stated in the beginning of this section, we will integrate the usage of Redis by modifying GET /user/:id. See this snippet below:

// index.js

app.get('/user/:id', async (req, res) => {
  let result = await redisConnector.get(req.params.id);
  console.log('Data from redis:', result);

  if (!result) {
    console.log('Failed getting data from redis, getting data from DB...');
    result = await mongoDbConnector.findOne(collection, {
      _id: ObjectId(req.params.id)
    });

    console.log('Data from DB:', result);
    if (result !== 'Data not found.') {
      const redisSetResult = await redisConnector.set(req.params.id, result, 3600);
      console.log('Set Redis result:', redisSetResult);
    }
  }

  res.send(result);
});

Before adding Redis, the function’s job is only to get the data by id from MongoDB. Now, we check if the data exists on Redis, then access the database if it is not. If the data exists in the database, set the data on Redis for future same requests.

First, we try to find the data from Redis from executing await redisConnector.get(). If the data is found, it skips the if, and then return result as the response.

But if the data is not found in Redis, we try to find the data on MongoDB. If the data exists in MongoDB, we also set the data in Redis for future requests by using await redisConnector.set(). And finally, we return the data from MongoDB as the response for GET /user/:id.

Test the code

Run npm start on your terminal and hit GET /user/:id. For testing purpose, get the id from your database. If it works correctly, it should show something like this:

On first hit:

On second hit:

Compare the SPEED difference

A simple way to check for improvements in speed after implementing Redis is by looking at the time needed to get the response. And even though it’s already fast in local machine operations, the requests go even faster after implementing Redis. See the screenshot below:

The first request was 132 ms and the second request was 11 ms. That’s more than 10 times of speed improvement!

If you repeatedly request the same user id, the time needed for getting the response will be around 11 ms, never reaching 132 ms again. Well, at least until the cache expires.

---

Let’s Refactor!

Remember dotenv from the first tutorial? You might notice there are currently so many variables for the services’ configurations in index.js. We don’t want this, because it violates the third rule of The Twelve-Factor App Methodology. So, let’s create our environment variables.

.env and .ENV.EXAMPLE

First, let’s create an .env.example file:

# .env.example

APP_HOST=localhost
APP_PORT=3000

DB_NAME=cil-rest-api
DB_HOST=mongodb://localhost:27017
DB_COLLECTION_USERS=cil-users

REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_EXPIRY_SECS=3600

There are three types of configurations here:

• General: APP_HOST and APP_PORT. These are the configurations for starting the application.
• DB: DB_NAME, DB_HOST, DB_COLLECTION_USERS. These are the configurations for connecting to the database.
• Redis: REDIS_HOST, REDIS_PORT, REDIS_EXPIRY_SECS. These are the configurations for connecting to Redis for caching.

Then, create file .env by copying from .env.example. Or, run this on your terminal: cp .env.example .env

index.js

Let’s modify index.js, where everything starts. On the very top of the file, import the dotenv library and get all the environment variables we set on .env before. See this snippet:

// index.js
const env = require('dotenv').config().parsed;
const config = {
    app: {
        host: env.APP_HOST,
        port: env.APP_PORT
    },
    db: {
        host: env.DB_HOST,
        name: env.DB_NAME,
        collections: {
            user: env.DB_COLLECTION_USERS
        }
    },
    redis: {
        host: env.REDIS_HOST,
        port: env.REDIS_PORT,
        expiry: env.REDIS_EXPIRY_SECS
    }
};

...

Then, we change the hard-coded values to getting from config. Here are the segments to change:

// index.js
...
const mongoDbConnector = new MongoDbConnector({
  name: config.db.name,
  host: config.db.host
});
mongoDbConnector.connect();
const collection = config.db.collections.user;

const redisConnector = new RedisConnector();
redisConnector.connect(
  config.redis.host,config.redis.port);

app.get('/user/:id', async (req, res) => {
  ...
  if (!result) {
    ...
    if (result !== 'Data not found.') {
      const redisSetResult = await redisConnector.set( req.params.id, result, config.redis.expiry);
      console.log('Set Redis result:', redisSetResult);
    }
  }
  res.send(result);
});

...

app.listen(config.app.port, () => {
  console.log(`cli-nodejs-api listening at http://${config.app.host}:${config.app.port}`)
});

The changes including when initiating MongoDbConnector, connecting to Redis, setting Redis expiry, and starting the app on app.listen().

OK, that should do it. Now, whenever you need to change the environment variables, change it on .env.

Test the Code

Just run npm start like usual. If everything runs like before doing the refactoring, then you are good.

Conclusion

This time we also covered quite a lot. Here are the things we have learned:

• Setting up Redis on local machine.
• Connecting to Redis server with Node.js REST API.
• Integrating the basic Redis operations to a route.
• Refactor the hard-coded variables to become environment variables.

And here’s the promised source code from GitHub!

Note that because a cache’s size is limited, it is not recommended to cache everything. To put it simply, you wouldn’t put everything on top of your working table, right? You will still need to put some less used things to the drawer.

Also, it is the best practice to put your environment variables separated from the source code. My personal rule when I have to define a value is by creating a constant. But, if it could change in different environments, or if it’s an indefinite variable, put it as an environment variable. After all, nobody wants to change the source code every time when deploying on different environments.

I hope you learn a lot this time! What are your experiences with caching, Redis, and environment variables? Let me know in the comments!

Next post in this series:
Part 4 – Improve Security with JWT

“Optimizing your strength is not optional, it’s an obligation.”

― J.R. Rim