Using rsync to backup

Question

I currently back up the contents of an external hard drive A by (1) deleting everything in the destination external hard drive B and (2) copying everything from A to B (in Finder). The external hard drive only contains “basic data” (folders, images, videos, documents, etc.), nothing fancy/weird.

But with all the amazing free tools that exist out there I couldn't find a decent one to backup (suggestions accepted).

I want to automize this. After a few searches, it seems I can use rsync. I want then to automize this by using rsync to avoid copying files that are already on the destination hard drive (of course, if a change has been made, I want to update the file/folder; the same with deleted files/folders or new files/folders; the idea is to mimic what I usually do manually with Finder).

I got to this point

rsync -av --progress --delete /Volumes/A/ /Volumes/B

Now the log is full of files which name starts with . or ._ or even ._.. Are they necessary?

Question

How can I copy only the files that matter, leaving all those that don't matter behind. For example, if I have photo.png I would expect to copy photo.png, not ._photo.png or many files.

What --exclude or --exclude-from can be used safely on a Mac? May be a good rule is “just copy anything that Finder shows” which is what I would copy if I passed manually through all folders.

 Extra question

Is -av --progress --delete enough? Am I committing a crime or risking some data? What options would you use.

Please, back your answer with some arguments, I would appreciate it.

I'm looking for a way to automate rsync which seems quite powerful and free to use.

Answer 1

For Maximum Safety, Copy Everything

The safest option is to copy everything, including invisible meta-data files.

Files tend to exist for a reason and as software changes, so will the existence, purpose, and contents of these meta-data files. Copying everything will reduce the maintenance burden and offer ease of mind that nothing is being lost.

The log file created by rsync will be technically involved by the very nature of rsync being a command line tool. Consider presenting a tidied up, meta-data filtered, and maybe even coloured coded output for the user to check. You could do this with a wide range of scripting tools and languages. Please ask more questions here on Ask Different and on StackOverflow if you need help with this approach.

Delete Last & Copy OS X Resources

Regarding the rsync flags, take a look at the question Fastest and safest way to copy massive data from one external drive to another. In this question, a few additional flags are used and explained:

sudo rsync -vaE --progress --delete-after /Volumes/SourceName /Volumes/DestinationName

In this situation, the E will ensure resource forks and other Mac specific properties are copied.

You may want to consider --delete-after to avoid deleting until the copy has completed; please note that this approach will potentially require a destination drive twice the size of the source.

Possible Exclusions

An answer to a related question, How can I omit FCPX Render Files from a Time Machine backup?, provided a useful link of OS X files and folders that can be excluded from most back-ups. This link provides a practical list of file patterns, folders, and paths that you could exclude.

Include dot Files

There are good reasons to back up files beginning with dots, .* matching files.

Some software keeps preferences, settings, and other information of value in folders at the top of the user's folder in invisible dot prefixed folders. Running ls -la ~/ will reveal these folders and files.

If any user uses or has software that in turn uses version control software, be sure to back-up dot files. Software like subversion and git both store critical information within their dot folders. These hidden folders can be scattered across your file system, where ever a project is checked out.

Spotlight and Disk Access

Spotlight is OS X's search service. Spotlight uses the mdworker process to index and update the search catalogue. If you are concerned about possible disk corruption or slow copies, disabling mdworker while running rsync may help. Personally, I leave Spotlight running while running large rsync transfers.

Answer 2

I'm not completely pleased with the current answers, but I will try to cover here a bit of the possibilities that I've seen on the web trying to find a nice setup for rsync.

And, by the way, if one is interested in Time Machine like copies, there's rsnapshot. And there's also Unison for two way syncronizations. Plus, there's actually a few GUIs, like Backup Utility and arRsync; not exactly what I was looking for but they might do the job for somebody.

First, my only intention was to duplicate photos and videos, so exact copy wasn't needed, hence no need to care too much. In fact most of my doubts were if I could exclude everything (the same that would happen if, say, I downloaded a photo from the internet, I just download a .png, everything else is not downloaded but autogenerated).

Here's an unstructured list of thoughts that you may want to take in account

• I you want to ensure that your laptop doesn't go to sleep, you may want to caffeinate the process caffeinate -s rsync -av .... Taken from here.

• If you are doing local copies, like in my case, or even if internet connection is not too slow, you should not use -z option (compression), and use -W (transfer whole file, rather than delta transfers; this is a default when local) and probably use --inplace to make transfers fast. Taken from here.

• You can use --delete-after so any file will be first transferred, and then moved on destination (and deleted the original one) and is safer than deleting before the transfer or during the transfer.

• You can stop the transfer by pressing ctrl + C, and it will stop cleanly. Taken from here. This was one of my fears with SuperDuper!, if you need to stop a transfer you get the message “You will leave the hard drive in an unknown state…”.

• In recent versions there's --info=progress2 which adds even more details to -v.

• There's -P (which equals --partial and --progress) that will leave mid-transferred files there so you can keep going when you restart the process (if for some reason you can't end the syncronization at once).

• One might be interested in stopping Spotlight or TimeMachine before doing the copy, and reenable them after the transfer. And even disable Spotlight for the external disk.

• Other options used in all those references include -x (or --one-file-system), -E (--executability), -H (--hard-links), -X (--xattrs), -A (--acls), and --sparse, --hfs-compression, --protect-decmpfs. You may want to look at them.

For me, a basic command looks like (I might use more options, but this is enough for an example):

caffeinate -s rsync -av --delete-after --progress --stats --exclude-from 'excludefile.txt' \
  "/Volumes/SrcHDD/" "/Volumes/DstHDD" # note the trailing slash / after the source

Now in my case I could just include {*.jpg,*.png,*.mp4,*.txt,*.pdf,…} and no one would say “you need also system files” but since I don't want to search to get all the possible filetype I have files of, I prefer to exclude. And there are things that not only can be but seem convenient to exclude.

I found a few links, take what you want:

1. https://bombich.com/kb/ccc4/some-files-and-folders-are-automatically-excluded-from-backup-task

2. https://gist.github.com/tvwerkhoven/4541989

3. https://github.com/necolas/dotfiles

4. https://gist.github.com/shkm/5531679

5. http://www.hackurmac.com/2014/08/backing-up-and-restoring-your-hackintosh.html

6. https://github.com/jedda/Counterpart

7. http://alanwsmith.com/rsync-exclude-list-for-mac-osx

And from there you could probably get what's safe or sane to ignore. Here's the full list (I just removed duplicates)

.HFS+ Private Directory Data*
/.journal
/.journal_info_block
.AppleDouble
.AppleDB
/lost+found

.metadata_never_index
.metadata_never_index_unless_rootfs
/.com.apple.timemachine.donotpresent
.VolumeIcon.icns
/System/Library/CoreServices/.disk_label*
/TheVolumeSettingsFolder

.DocumentRevisions-V100*
.Spotlight-V100
/.fseventsd
/.hotfiles.btree
/private/var/db/systemstats

/private/var/db/dyld/dyld_*
/System/Library/Caches/com.apple.bootstamps/*
/System/Library/Caches/com.apple.corestorage/*
/System/Library/Caches/com.apple.kext.caches/*

/.com.apple.NetBootX

/Volumes/*
/dev/*
/automount/*
/Network/*
/.vol/*
/net

/private/var/folders/zz/*
/private/var/vm/*
/private/tmp/*
/cores/*

.Trash
.Trashes

/Backups.backupdb
/.MobileBackups
/.MobileBackups.trash
/.MobileBackups.trash

Library/Mobile Documents.*
.webtmp

/Library/Caches/CrashPlan
/PGPWDE01
/PGPWDE02
/.bzvol
/Library/Application Support/Comodo/AntiVirus/Quarantine
/private/var/spool/qmaster
$Recycle.Bin
Saved Application State
Library/Preferences/ByHost/com.apple.loginwindow*

.AppleDesktop
.com.apple.timemachine.supported
Network Trash Folder
/Library/Caches/*
/private/var/spool/postfix/*
$RECYCLE.BIN
/stuff/*
/afs/*
/private/var/run/*
/Previous Systems.localized
/tmp/*
.dbfseventsd
.DS_Store
.PKInstallSandboxManager
.SymAV*
.symSchedScanLockxz
.TemporaryItems
Desktop DB
Desktop DF
hiberfil.sys
pagefile.sys
Recycled
RECYCLER
System Volume Information
Temporary Items
Thumbs.db
AUTORUN.INF
Autorun.inf
autorun.inf

Answer 3

It's August, 2022; This is an updated answer:

Before getting into the answer here, you must know that although an rsync binary is included with macOS, it is now an ancient, 16-year-old software package. On my Catalina system /usr/bin/rsync --version reports itself as ver 2.6.9; released in Nov, 2006. The rsync release history since ver 2.6.9 seems to make it clear that Apple's decision to halt upgrades for rsync on their OS was based entirely upon the rsync team's decision to switch to the GPLv3 license. This is unfortunate because rsync ver 3.0 (the release following 2.6.9) included significant upgrades specifically supporting macOS users:

• --xattrs : preserves OS X extended attributes
• --acls : preserves OS X Access Control Lists
• --fake-super : allows non-admin users to preserve all attributes

The point of the above paragraph is simply this: If you want to use rsync to back up files on your macOS, you must have a later version than the one Apple provides. Otherwise, you will lose all xattrs and ACLs - metadata that plays a significant role in several features of macOS. This is easily verified with a simple experiment.

Fortunately, current versions of rsync are available from 3rd party macOS package managers (MacPorts for example), or for the DIY-inclined, you can clone the rsync GitHub repo and build it from source.

It's also worth noting that a current version of rsync can repair/restore a backup made previously using Apple's native rsync ver 2.6.9. It can also repair/restore xattrs, ACLs and date-timestamps that have been mangled or lost by Apple's "Migration Assistant".

rsync is a mature & comprehensive package with numerous options. Many of these options are not obvious choices, and one can spend a fair amount of time developing an option set to meet specific objectives. While there are no real shortcuts to careful consideration of your own use-case, I'll suggest the following option sets as a reasonable generic rsync command for general use on macOS - a "point of departure" in other words. You may further tailor these to meet your own needs.

I'll present two cases for using rsync to backup and restore; one using the --delete option, and one without. Both cases assume the local Macintosh HD is the source for the backup operation, and the destination for the restore operation. In my own case, there are some folders (e.g. Documents) for which I prefer to use the --delete option, and other folders (e.g. Music, Pictures & Movies) in which I do not use --delete. This is a personal choice that reflects how I manage/organize my data.

In the context of rsync, the --delete option simply means that deletions you make on your source folders are propagated to your destination folders.

Without the --delete option:

1. In a BACKUP operation:

i. Source: /MacHD_Folder/

ii. Destination: /mountpoint/for/remote/storage

rsync -rlAXtgoDiv --fake-super /MacHD_Folder/ /mountpoint/for/remote/storage

2. In a RESTORE operation:

i. Source: /mountpoint/for/remote/storage

ii. Destination: /MacHD_Folder/

rsync -rlAXtgoDiv --fake-super /mountpoint/for/remote/storage/ /MacHD_Folder 

With the --delete option:

1. In a BACKUP operation:

i. Source: /MacHD_Folder/

ii. Destination: /mountpoint/for/remote/storage

rsync -rlAXtgoDiv --fake-super --delete --backup-dir=/mountpoint/for/remote/storage/deletes /MacHD_Folder/ /mountpoint/for/remote/storage

2. In a RESTORE operation::

i. Source: /mountpoint/for/remote/storage

ii. Destination: /MacHD_Folder/

rsync -rlAXtgoDiv --fake-super /mountpoint/for/remote/storage/ /MacHD_Folder 

Note the asymmetry between backup & restore operations when using the --delete option. This due to the fact that nothing need be deleted for the restore operation. Note also use of the --backup-dir option when using the --delete option: This causes rsync to move all deleted files & folders to the designated folder on the destination drive. You may think of this as a form of "insurance" for deleted files - it functions as the Trash folder does on your local MacHD.

Further Reading:

This GitHub recipe offers further explanations and useful options.

Answer 4

If you're using rsync version 3.0.6 as per Carbon Copy Cloner or 3.1.2 as per Homebrew, you can take a cue from Carbon Copy Cloner arguments:

rsync -A -X -H -p --fileflags --force-change -l -N -rtx --protect-decmpfs --numeric-ids -go --delete-during --backup --backup-dir=</PATH/TO/STICK/BACKUP_when_using_delete> --protect-args <SRC>/ <DEST>

Answer 5

I'd advise against pruning meta-data during a backup, particularly the dot-files e.g. ._$filename, however if you really want to exclude the dot-files from your rsync command add --exclude '.*' to it.

rsync -av --exclude '.*' --progress --delete /Volumes/A/ /Volumes/B

Answer 6

I have used rsync for backups at several jobs, and I use it at home.

I highly recommend it, but with some modifications. As a backup tool, it's great, but as an archiving tool it falls a little flat. Yes, it copies everything, but you don't get versions of everything, you always get the latest versions only.

I used this guide http://www.mikerubel.org/computers/rsync_snapshots/ as a jumping-off point. Read the whole page. It does a great job of explaining the options, and outlines how you can implement incremental backups. And amazingly, the article is over 10 years old but is still applicable today. Gotta love unix.