1

I have an old hard drive and basically clone it to a new hard drive using Grsync or Timeshift. Because the older hard drive is likelier to fail, I'd like to use the newer one as my in-use hard drive and the older one just as my backup drive where all changes are synced.

First asked about this at the Vorta GUI repo since ultimately I'd like to use only one GUI for all backup tasks. Because that tool doesn't (yet) have this functionality, I'd like to use another open source GUI. Further differences to related questions and to what some available tools I tried offer is described below:

  • Syncthing: For strange reasons it's not really supporting backups to local destinations (syncing local hard drives or folders); my second hard drive is local
  • Unison: I'd like the syncing to be always-on automated synchronization without any triggering needed
  • Rsync GUIs: These seem to have the same problem as Unison (recommended here): I'd like to have a directory monitor daemon / File System Monitor that detects changes on the large HDD I use as they are made and then at some point (immediately or specified intervals) syncs them to the second hard drive automatically. With tools like Grsync ("Delete on destination" option enabled) or BackInTime (unmaintained) this can only be done manually. I'd like the files to be exactly the same so I could mount and use the fully-encrypted drive exactly as if it was the old one if the new one fails (for example file permissions and everything else should stay the same). It should be efficient and for example not scan the whole drive for changes anew every time.

To me this seems like the most intuitive way desktop users could backup their full hard drives without versioning so I was surprised it's not very easily possible with most existing GUI tools. Even worse, other questions (related but not the same!) such as this seem to be only about command-line solutions for niche applications techies and even without a concrete rsync command one could use or information how to implement it. I use Debian with KDE.

mYnDstrEAm
  • 338
  • I can only think of a somewhat involved scripting solution to this, given the requirement for it to be either immediate or at a later time (when the 2nd disk gets plugged in). The basic idea is that you stand up an inotifywatch on the primary's mountpoint and either trigger the sync of each file/directory as it receives a write_close event (note that a recursive watch won't pick up changes in newly created directories, so you'd have to restart the watch when a new dir gets created) or write the name to some tempfile and have it synced by a mount-script. – tink Jan 01 '24 at 19:53
  • @tink I'd prefer it only syncs at certain intervals because then the second disk can be in standby most of the time which may be good for its lifespan, the main reason is that I'd like it to be silent so if it syncs all the time it'll be too loud than if it was doing that only every few hours. There must be a tool for that - I thought this was the most reasonable way to backup for 90% of people if you either don't need versioning or want to have nonprimary HDD (such as an external HD) get backed up. The dirmon would simply log which files have been modified or changed and then let it back up. – mYnDstrEAm Jan 01 '24 at 20:49
  • 1
    I'm not convinced that it "being on stand-by" is the best thing to assure longevity. Most HDD failures I've encountered in my career occured when a machine got rebooted, AKA the spindle stopped and restarted. But yes, you could use inotify to just generate that list and then act upon it say every thirty minutes if that's what you desire. – tink Jan 01 '24 at 20:52
  • In addition to what tink said... You're now saying the opposite of what you had said in your post – you specifically excluded Unison and Rsync because they weren't continuous, and now you don't want continuous sync. If your goal is to only sync at certain intervals (which literally means being triggered by a scheduled timer), then what are the actual issues with using e.g. Unison? – u1686_grawity Jan 01 '24 at 20:57
  • @u1686_grawity No, it's not the opposite and I never said I wanted continuous sync. I'd like to monitor all file changes and automatically sync at intervals if possible (continuously may be an option too however). Unison doesn't have a a filechanges monitoring daemon and doesn't have a way to configure automatic syncs at intervals. – mYnDstrEAm Jan 01 '24 at 21:03
  • Sorry for the close vote, but I believe this is off-topic for 2 reasons - First it amounts to a software recommendation request, and second it appears to be an XY question - you seem to be assuming the approrpiate solution in your last paragraph. – davidgo Jan 01 '24 at 21:10
  • Assuming the question is not closed you may want to modify it - the title says "two local hard drives in sync", but most of your post if about keeping files in sync. These are 2 very different things. Keeping drives/block devices in sync is normally done at or below the filesystem level - eg using software raid or LVM (or drbd!) - while versioning files is normally done with something like Rsync - although ZFS blurs a lot of lines. – davidgo Jan 01 '24 at 21:13
  • @tink - I solved this problem many, many years ago using DRBD and loopback - See https://davidgo.gottschen.com/2012/09/30/drbd-for-ssdusb/ if this kind of thing interests you. – davidgo Jan 01 '24 at 21:17
  • @davidgo Good that you explain it and not just vote. This question is asking how to do that with the requirement that a GUI is used for that rather than a CLI. I'm just specifying what the requirements are and not "assuming the approrpiate solution" in the last paragraph. No, I want to keep two hard drives in sync (or two partitions as each of those have one partition)...the best way for that may be keeping the two directories in sync but I'm leaving that open to the answers and would just prefer the most efficient (note: they're encrypted). Also I said I don't need versioning. – mYnDstrEAm Jan 01 '24 at 21:23
  • Sorry, but "you are doing it wrong". I expect you "don't want to hear it" but the solution to your problem is to rebuild your system so it uses RAID1. RAID is the definition of what you seem to be wanting to achieve - ie an immediate clone of data onto a second drive to protect against failure". It is often possible to convert a drive to RAID, but it would be considered high risk and require good to excellent command line skills. Many distros allow you to set up RAID through a GUI as part of a fresh install though. – davidgo Jan 01 '24 at 21:26
  • Actually, MAYBE, if you are very, very lucky, you can do some hackery with LVM - what is your output of "sudo /bin/bash"; "pvs; vgs; lvs" (sorry, you need to do it from a terminal window) – davidgo Jan 01 '24 at 21:28
  • See: that is one of multiple ways this could be solved. But from what you wrote it seems like there is no GUI tool for converting the HDD to RAID1 and in addition comes with data loss risk. I don't get the Close votes; you're just voting to censor a valid question. – mYnDstrEAm Jan 01 '24 at 21:38
  • Let us continue this discussion in chat. – davidgo Jan 02 '24 at 01:59
  • 1
    Try to use FreeFileSync with a scheduled task, as described here: https://freefilesync.org/manual.php?topic=schedule-batch-jobs There could be issues though with system files which are access protected during execution, so I'd run a second batch job on shutdown for all those files. – 1NN Jan 08 '24 at 08:24
  • None of your suggestions clone a disk; they copy the filesystem contents. Mostly the same result for an end user, but definitely not a clone. – Chris Davies Feb 03 '24 at 18:01

1 Answers1

1

Here's the best I came up with so far and I'll edit it if I find a better way:

  • Option #1: Convert the disk to RAID1 - The three problems with that are that:

    • If you delete something on the source disk it will be deleted on the second as well but for desktop use-cases it's better if you can then restore the directory or file for some time until the next backup (this is useful for servers where this can avoid downtimes but not good for personal backups)
    • There are quite high chances you'll lose your data so this should only be done if you backed up your data on yet another third drive
    • There are some step-by-step tutorials for that (where it's difficult to see which is up-to-date and fits your requirements) but no GUI

  • Option #2: Using Grsync. This seems to be the only advanced GUI for rsync that is still developed since BackInTime and LuckyBackup are not actively developed anymore. I haven't checked if the Timeshift GUI can do what I'm trying to as well and there are probably some less developed GUIs for rsync that can do the same. These options are enabled for syncing the specified directory which is the whole drive (changes from default settings; please correct any that are wrong):

    • "Delete on destination"
    • "Preserve owner"
    • "Preserve group"
    • "Preserve timestamp"
    • "Preserve permissions"
    • "Copy symlinks as symlinks" in the Advanced tab, fourth from top on the left; if you use shortcuts for files this ensures they are also there in the the backup directories
    • ("Verbose" is disabled so it doesn't produce large logs)
    • ("Only compare file-size" may make it faster but I haven't tested this; disabling "Show progress" may be useful too)

    One then simply has to manually regularly start the program with the saved profile and click on the play button to refresh the sync backup. Save the profile by going to Session->Export so you can use always the same backup profile. The problems with this solution are:

    • It can't do scheduled or automatic backups. I would submit a feature request about it (linked here) but it seems like it doesn't even have a modern git repo.
    • If you also want your root permission files synced you still need to run a command after the sync run since there is no root GUI for Grsync (there is one for BackInTime): in Grsync go to File->R sync command-line, copy the command into a terminal but add sudo in front of it before running it (as the root user). Also remove --progress there if you have that enabled. For me the command is sudo rsync -r -t -p -o -g --delete -l -s dir1 dir2 where dir1 is just the root directory of disk1. (Instead of using Grsync you could also just use the commandline until there is at least one GUI that allows for all functionality.)
    • I don't know how well suited it is for syncing large whole drives. Since this question may get closed this may remain unclear. I would imagine that a file watcher type of daemon that logs which files you have changed as you change them would make it much more suitable for that and I don't think rsync/Grsync has that functionality. The same applies to when you only move files around where it may delete them in the destination to copy them over another time rather than just moving the folder or files there.
mYnDstrEAm
  • 338
  • I feel you might mix up two separate concepts: resiliance against hardware failure and have backup history to undo something. Raid is against hardware failure and seems exactly what you want with your old drive. But you ALSO want pointwise backup to (at best: off site) media, which is something else! Good esp. when you need a history of past snapshots. Think a minute about how differnt a restore process for each would be: on failing hardware or for a need to restore past snapshot. You need both? Then separate things. For using raid1 I actually would try to convince you against gui ;-). – tverrbjelke Jan 11 '24 at 12:30
  • No, I'm not looking for a backup history and made that clear in the question. Synchronizing already means you don't have backup revisions. I explained why Raid currently can't be the solution: mainly no GUI and data loss risk (once these two issues are solved it could be the better answer here). I don't want a history of snapshots and can't have one due to disk space. Restore process: one drive fails, I use the other until I have one new drive. GUI is absolutely required and it's completely precluded if there is no such, and it will never be widely used by the general population without a GUI. – mYnDstrEAm Jan 11 '24 at 13:06
  • Your rsync suggestion could be simplified to sudo rsync --archive --delete dir1 dir2 – Chris Davies Feb 03 '24 at 17:59