34

I was given the task to set up incremental backups for MongoDB replicaset, as start point, of course, I googled it and could not find anything on MongoDB docs, I did find however this question on Stack Overflow, which encouraged to develop my own solution as didn't find Tayra very active.

I read about oplog and realized it was very easy to develop something to replay the log, but it turns out that I didn't have to as mongorestore does that for me.

Now I have a working solution with bash scripts and it was quite easy, that's the reason I am asking here if there is any flaw in my logic, or maybe something that will bite me in the future.

Below how I implemented that:

Full backup procedure

  1. lock writes on a secondary member db.fsyncLock()

  2. Take snapshot

  3. Record last position from oplog

    db.oplog.rs.find().sort({$natural:-1}).limit(1).next().ts

  4. Unlock writes db.fsyncUnlock()

Incremental backup procedure

  1. lock writes on a secondary member

  2. Dump oplog from the recorded oplog position on full (or latest incremental ) backup:

    mongodump --host <secondary> -d local -c oplog.rs -o /mnt/mongo-test_backup/1 
    --query '{ "ts" : { $gt :  Timestamp(1437725201, 50) } }'

  3. Record latest oplog position (same way as for full backups)

  4. Unlock writes

Full backup restore procedure

  1. stop all instances of mongod
  2. copy snapshot to data dir of the box which will be the primary, but make sure to exclude all local* and mongod.lock this restore technique is called reconfigure by breaking mirror
  3. Start primary
  4. reconfigure replicaset
  5. start secondaries without any data, let them perform the initial sync. Or copy the data from the new primary with fresh local database

Restore incremental backup

When we created incremental backup it stored it like this:

/mnt/mongo-test_backup/1/local/oplog.rs.bson
/mnt/mongo-test_backup/1/local/oplog.rs.metadata.json

We're instered on oplog.rs.bson but we will have to rename it, so here are the steps:

  1. change directory to the backup: cd /mnt/mongo-test_backup/1/local

  2. delete the json file rm *.json

  3. rename the bson file mv oplog.rs.bson oplog.bson

  4. restore it:

     mongorestore -h <primary> --port <port> --oplogReplay /mnt/mongo-test_backup/1/local

I have it all scripted, I may commit it on GitHub later.

The question is if there is any flaw in the logic? I am a bit suspicious as the procedure is quite straight forward and still I couldn't find it documented anywhere.

Community
  • 1
Tiago
  • 441
  • 1
  • 4
  • 3
  • 3
    What version of Mongo are you using? If you are using wiredtiger, then the first item you referenced with db.fsyncLock() is a problem. MongoDB Inc claims "With WiredTiger, the fsync command with the lock option cannot guarantee that the data files do not change. As a result, do not use these methods to ensure consistency for the purposes of creating backups." link – SDillon Jul 24 '15 at 12:29
  • 1
    @SDillon using 3.0.4 but not using WiredTiger, at least not yet. I we decide to use it, instead of lock writes we will have to stop mongod all together. It's a fair point thanks – Tiago Jul 24 '15 at 12:35
  • I found the following tool for incremental backup https://github.com/EqualExperts/Tayra hope this will help – Ahmad Abuhasna Jan 19 '16 at 10:05
  • 1
    "Changed in version 3.2: fsync command with the lock option can ensure that the data files do not change for MongoDB instances using either the MMAPv1 or the WiredTiger storage engines, thus providing consistency for the purposes of creating backups." – safety Jan 10 '17 at 08:39
  • Normal (and absolutely easiest) way to do incremental backups is use LVM and snapshots. https://docs.mongodb.com/manual/tutorial/backup-with-filesystem-snapshots/ – JJussi May 05 '17 at 14:24
  • Respectfully disagree. In the link you provide is clearly stated "No Incremental Backups

    This tutorial does not include procedures for incremental backups. Although different snapshots methods provide different capability, the LVM method outlined below does not provide any capacity for capturing incremental backups."

     – TanisDLJ Sep 04 '17 at 14:08
  • Hi @Tiago, just checking if you pushed your solution to Github? – Awanish Raj Jun 22 '18 at 08:38
  • @AwanishRaj unfortunately not, we didn't spend enough time to make it re-usable outside that client, but the steps outlined above work very well, we had to restore it a couple of times and all went smoothly. The challenge in making it reusable was not related to Mongo itself but the restore coordination, which requires passwordless ssh setup and sudo privileges, so it became a chef cookbook instead of shell script. – Tiago Jun 22 '18 at 20:57

1 Answers1

5

To answer your question. No! There is no fail on your logic and it should work without problems. However, if LVM snapshots can be used, it's better way to do backups.

JJussi
  • 5,703
  • 1
  • 15
  • 19
  • How do you do incremental backups of an LVM snapshot? Thanks! – TanisDLJ Sep 04 '17 at 12:56
  • LVM snapshots are incremental by nature. Snapshot are moments in time and there are only changes recorded. – JJussi Sep 04 '17 at 13:54
  • Just a snapshot taken yes, it is incremental. But if you archive the snapshot then is a full backup. You cannot archive different incremental backups like duplicity does, for instance. And you cannot simply start creating snapshots every 30 min for incremental backups as it would hit the performance really bad. – TanisDLJ Sep 04 '17 at 14:06