engineering

MongoDB Replica set, Oplog and other ghosts that haunt you.

Okay, so this comes after two frustrating days of reading documentations and finally getting the application in running state.

Chaitanya Giri 19-10-2019
Share this

 

What is a replica set?

As you might have guessed from the name it is a replica of the data set. Every now and then we suffer data loss and this can be crucial when it comes to huge data that are essential for a company to even exist. Suppose next day you wakeup and open amazon to buy something and you do not find a single product 😆. Now this is very less likely to happen because we have replica set. (Note: Amazon do not use mongodb to store data this was an example, though they use a NoSQL database so it has to be something similar).

Replica set has been in mongodb for a while now, previously instead of replica set we had master-slave replication(Please refer some other article if you want to learn about this).

How does it work?

Basically a replica set is multiple set of mongod instances that hosts same data(this can be tricky to say as in any instance some data might be put after some delay but for the moment let’s assume as it is).

Now assume we have a primary node that has multiple secondary nodes(max 50) say 2 in this case. When ever client writes or reads from primary node the primary node logs it in its oplog. Now the two secondary nodes replicate the oplog’s operations and in-turn has the same data as primary. Now this nodes are connected with the primary node with a heartbeat kind of structure in if the primary node does not reply to their heartbeats in 10 seconds. Secondary nodes do an election sort of thing to replace primary node(😢 So rude). Now if there are less than 7 secondary nodes and total number of secondary node is even then there is an extra node called Arbiter it do not replicates it simply participate in election so that in the election a draw among nodes never happens and if there are 7 or more nodes then any 7 will participate based on their priority value(more details ahead).

Secondary nodes also have a priority value attached to them which helps in election. The extract of this statement can be: If a node has priority value 0 then it will never be elected and it’ll always have the data with it so it provides an extra layer of security to data loss.(Amazon is safe 😆).

Now how can you actually setup a replica set in MongoDB?

  • By default, MongoDB bind to local interface only, it will restrict the remote connections. So bind_ip option tells MongoDB to accept connections from which local network interfaces. Below command specifies the replica set name and the ip binding through the --replSet and --bind_ip command-line options.
mongod --replSet "rs0" --bind_ip localhost,<hostname or ip address of mongod host>

Now you have done the replication part you need to do is set up MongoDB Oplog Tailing.

Oplog Tailing means the application server will be notified real-time of any changes happening in the entire database.

In Meteor applications, when we talk about “oplog tailing” this means that the Meteor app is aware of this oplog, and watches it for changes, reflecting those changes in Meteor when those changes occur. In Meteor app, oplog tailing is turned on by default in our development environment, but requires a little extra work to get set up in our production environments.

MONGO_URL='mongodb://app_user:app_passwd@127.0.0.1:27017/app_db'MONGO_OPLOG_URL=mongodb://oplog_user:oplog_passwd@127.0.0.1:27017/local?authSource=admin

Note: You need to create app_user and oplog_user and assign roles before previous step.

  • To create Oplog user :
use app_dbdb.createUser({user: "oplog_user", pwd: "oplog_passwd", roles: [{role: "read", db: "local"}]})
  • To create App user :
use app_dbdb.createUser({ user: "app_user", pwd: "app_passwd", roles: [{ role: "dbOwner", db: "app_db" }] });
Subscribe to newsletters
Get the updates right in your inbox. We never spam!

Read next