News for hackers. Stuff that matters.
Re:Ask HN: Can we talk about FreeBSD vs. Linux? by laumars on Saturday November 22, 2014 @02:02PM (#8646322)
That's called copy-on-write (CoW) and is supported by a few modern filesystems (including ZFS). In fact ZFS does do some very basic deduping in the way you suggest (ie you copy a file instead of move it, and ZFS will just issue a pointer).

However full deduplication could never be free simply because of the overhead of keeping a table of all the duplicated data and scanning new content for duplications.

Re:Ask HN: Can we talk about FreeBSD vs. Linux? by byuu on Saturday November 22, 2014 @02:51PM (#8646457)
And ZFS deduplication consumes an absolutely frightening amount of RAM: 5GB of RAM per 1TB of storage is recommended.
Re:Ask HN: Can we talk about FreeBSD vs. Linux? by laumars on Saturday November 22, 2014 @03:04PM (#8646503)
Oh totally. ZFS gets a lot right but even ZFS can't make deduplication practical for all bar a few fringe cases.
Re:Ask HN: Can we talk about FreeBSD vs. Linux? by badgersandjam on Saturday November 22, 2014 @03:12PM (#8646526)
That's not much. Our baseline config HP DL380s come with 64GiB RAM (!!)
Re:Ask HN: Can we talk about FreeBSD vs. Linux? by KaiserPro on Saturday November 22, 2014 @05:41PM (#8646925)
File servers should really have Looooots of ram. after all its cheap cache (compared to a fusion io card) Our file servers have 384 gigs of ram. The next gen will probably have much more.
Re:Ask HN: Can we talk about FreeBSD vs. Linux? by the_why_of_y on Sunday November 23, 2014 @08:12AM (#8648687)
Copy-on-write is not the same thing as content-addressable.

ZFS has deduplication as an optional feature, which is implemented as a content-addressable store of filesystem blocks. In contrast to e.g. git the content-addressable aspect is an implementation detail that is not exposed to users.

http://blogs.sun.com/bonwick/entry/zfs_dedup

Re:Ask HN: Can we talk about FreeBSD vs. Linux? by laumars on Sunday November 23, 2014 @06:00PM (#8650284)
> Copy-on-write is not the same thing as content-addressable.

I know that. But CoW does cover a few points raised by the previous poster.

> ZFS has deduplication as an optional feature, which is implemented as a content-addressable store of filesystem blocks. In contrast to e.g. git the content-addressable aspect is an implementation detail that is not exposed to users.

I know what dedup is and how ZFS utilises it (I've been running ZFS for about 8 years now - I'm quite familiar with it).

If you read my post again, you'll see I was discussing two separate points: 1) that CoW file systems do provide the pointer-like methods the former commenter raised. And 2) deduping isn't free.

What you're arguing with me is semantics and if you read the former post again, you'll understand why I chose the language I chose.