News for hackers. Stuff that matters.
Re:Time-Machine-style backup with rsync by quesera on Sunday October 27, 2013 @09:21PM (#6624521)
The key feature of Time Machine is hard links to directories -- which is only possible on modern HFS+ (and rsync doesn't even try). Some people like the UI too, of course.

Without hard linked directories, a full --link-dest backup of a decent sized disk, with zero file changes from the previous pass, can easily consume 100MB (and take 45 minutes to perform).

This disk consumption might seem insignificant, today, but that's 2.4GB per day if you run a standard Time Machine equivalent backup schedule. Of course you might not choose to do that, because the previous hour's backup would only finish 15 mins before the next one started, which is insane.

These numbers are from direct experience on a 2TB, approximately 60% utilized source drive.

That said, I use rsync, not Time Machine, for my OSX backups. You'll want a few additional switches for HFS+, and if your target drive is HFS+ also, make sure you turn OFF "ignore ownership on this volume" in the Finder... but the script posted here has the right general idea. Somewhere on my project list is adding directory hardlinking to rsync.

Re:Time-Machine-style backup with rsync by laurent123456 on Sunday October 27, 2013 @11:07PM (#6624877)
The --link-dest option of rsync, used in this project, actually creates hard links.
Re:Time-Machine-style backup with rsync by quesera on Monday October 28, 2013 @09:04PM (#6631090)
...to files, but not to directories, which is the key point.
Re:Time-Machine-style backup with rsync by vvhn on Monday October 28, 2013 @12:43AM (#6625186)
the hard linking of directories is a hack not really a feature. There's a good reason why filesystems ( including HFS+ when not being used for Time Machine) do not do it.
Re:Time-Machine-style backup with rsync by wtallis on Monday October 28, 2013 @01:33AM (#6625366)
It's a hack, but it's also the best (only?) way to deduplicate all that metadata without breaking the ability to access the history using pre-existing filesystem operations.
Re:Time-Machine-style backup with rsync by oofabz on Monday October 28, 2013 @02:58AM (#6625593)
Volume snapshots are a better way. The only filesystems I know of that support them are ZFS, BtrFS, and HAMMER. I'm not sure if snapshots are implemented using hard links or if they are more fundamental.

Another way, which is worse, is block-level deduplication. All of the above filesystems support it, as does NTFS.

I wish Apple would adopt HAMMER for Mac OS. It is BSD-licensed and more suitable for a memory-constrained environment than ZFS.

Re:Time-Machine-style backup with rsync by wtallis on Monday October 28, 2013 @02:51PM (#6628975)
I'm not aware of any volume snapshot system that presents the history as a plain directory tree. At best, you can use some special mount options to mount a snapshot, but you generally have to use a special tool particular to that filesystem in order to access a snapshot, and creating the snapshot always requires such a tool. There's no standard Unix way to create or access volume snapshots. Time Machine histories can be created, accessed, and analyzed entirely with standard tools except for the modified `ln`.
Re:Time-Machine-style backup with rsync by Tobu on Monday October 28, 2013 @06:24AM (#6626147)
Btrfs snapshots are implemented using COW-shared trees (multiple trees for data and metadata). They can't be implemented with hardlinks, that would be a very leaky abstraction.
Re:Time-Machine-style backup with rsync by e12e on Monday October 28, 2013 @05:13AM (#6625905)
Hm, how about nilfs2?

http://www.nilfs.org

Re:Time-Machine-style backup with rsync by beagle3 on Monday October 28, 2013 @06:28AM (#6626161)
Why is block level deduplication worse?
Re:Time-Machine-style backup with rsync by fulafel on Monday October 28, 2013 @07:07AM (#6626260)
Why would deduplicating the metadata be important for this use case? Static amount of available inodes on some filesystems is the only reason I can think of.
Re:Time-Machine-style backup with rsync by wtallis on Monday October 28, 2013 @02:56PM (#6629019)
Because of the aforementioned problem where it takes 100MB and the better part of an hour just to record a new snapshot when literally nothing has changed since the previous snapshot. Deduplicating gets rid of that 100MB overhead, and being able to explicitly do it with hard links instead of having to rely on the filesystem to discover the duplication on its own takes care of most of the running time overhead.
Re:Time-Machine-style backup with rsync by porker on Monday October 28, 2013 @04:27AM (#6625782)
(How) would AuFS fit into this? I've only come across it with Docker.io, but it sounds like it handles incremental backups?
Re:Time-Machine-style backup with rsync by gizmo686 on Sunday October 27, 2013 @10:19PM (#6624713)
What is the benefit of hard-linking directories relative to soft-linking them?
Re:Time-Machine-style backup with rsync by aaronbee on Monday October 28, 2013 @12:29AM (#6625141)
The problem with soft-links is that if the underlying file/directory is deleted, you are screwed. For example if you have 100 full-machine backups and want to free some space so you decide to delete every other one, you have to be careful that none of the backups you are keeping, have soft-links to files in the backups you are deleting.

With hard-links the underlying data is not deleted until all hard-links are deleted, so you can delete any individual backup directory without losing data in any other backup directory.

A soft-link is like a pointer in C whereas a hard-link is like a C++ shared_ptr, ie. reference counted.