News for hackers. Stuff that matters.
Re:What’s New in Apple Filesystems [pdf] by jandrese on Tuesday June 11, 2019 @01:31PM (#20157562)
My understanding is that deduplication has generally been not worth the effort for a general use filesystem. The cost in memory, cpu, and sheer complexity doesn't pay off in enough storage savings. Disk space is relatively cheap. Filesystem crashes/corruption are not.
Re:What’s New in Apple Filesystems [pdf] by vardump on Tuesday June 11, 2019 @01:35PM (#20157585)
Yeah, block level deduplication is not useful for files on a typical consumer laptop. I can imagine it's even less useful for files on iOS devices.

However it is often great for virtual machines and other software development related things, as long as all the underlying block sizes and alignments match.

You really do want strong integrity guarantees though. One corrupted block can cause a lot of damage.

Re:What’s New in Apple Filesystems [pdf] by Scaevolus on Tuesday June 11, 2019 @02:04PM (#20157894)
The filesystem encryption used on iOS probably makes deduplication even less effective.
Re:What’s New in Apple Filesystems [pdf] by vardump on Tuesday June 11, 2019 @02:20PM (#20158068)
Depends on the layering order. You can do both dedup and encryption in the same time, if the encryption occurs on a lower layer.
Re:What’s New in Apple Filesystems [pdf] by eridius on Tuesday June 11, 2019 @02:25PM (#20158125)
Encryption in iOS is per-file. I'm not sure the OS really wants to go ahead and individual decrypt large numbers of files just to check if there are any duplicated blocks.
Re:What’s New in Apple Filesystems [pdf] by Damogran6 on Tuesday June 11, 2019 @01:55PM (#20157788)
Not only that, but a Server with 100 VM's is going to have a substantial percentage of each OS image as identical...not so much on an average consumer laptop.
Re:What’s New in Apple Filesystems [pdf] by FPGAhacker on Tuesday June 11, 2019 @01:40PM (#20157639)
Disk space isn’t cheap on a Mac laptop.
Re:What’s New in Apple Filesystems [pdf] by mikepurvis on Tuesday June 11, 2019 @01:46PM (#20157703)
Yeah, but that's a customer cost, not an Apple cost.

In any case, dedup just makes things more confusing to the customer— like, "you have X GB in files on your computer, but more than N-X GB free space because of magical deduplication." Or worse, "You deleted all those files, but didn't actually free up any space because they were dupes. So sad."

Re:What’s New in Apple Filesystems [pdf] by Wowfunhappy on Tuesday June 11, 2019 @01:56PM (#20157799)
> In any case, dedup just makes things more confusing to the customer— like, "you have X GB in files on your computer, but more than N-X GB free space because of magical deduplication." Or worse, "You deleted all those files, but didn't actually free up any space because they were dupes. So sad."

For what it's worth, this can already happen in APFS. While they don't have real deduplication, if you copy a file within Finder, that copy won't take up extra space.

Re:What’s New in Apple Filesystems [pdf] by jjoonathan on Tuesday June 11, 2019 @03:28PM (#20158715)
How granular is the COW? If you touch a byte, does just a block get copied or the whole file?
Re:What’s New in Apple Filesystems [pdf] by cmurf on Tuesday June 11, 2019 @04:53PM (#20159441)
It depends on the application. Often applications do a kind of COW on their own, behind the scene they write out a new file with a randomized naming scheme, when that succeeds, do a rename/replace. So in this case you're getting a new file, as in, a file with a new inode.

If your application is smarter, then it could just COW one block, whatever the minimum block size is for that file system. e.g. for Btrfs it's page size, so x86 that's 4KiB, ppc64le it's 64KiB. I'm not sure what it is for APFS, I'd guess 4KiB. It could actually be inefficient depending on the use case, e.g. a 1MiB file with a single byte change in the middle, if it were a single 1MiB extent, becomes three extents with COW:

    0-500KiB - extent 1
    500-504KiB - hole where stale data is
    504KiB-1MiB - extent 3
    4KiB   - extent 2, with new data, a copy of 4095 bytes from the above hole, with 1 bytes of new data.
Edit: 2nd paragraph suggests the application would COW if it's smarter; that's not correct. If it's smarter, it would request an overwrite of some portion of the file and the file system is what would COW (if it's a COW file system). Granularity depends on a combination of application and file system.

Edit 2: More detailed answer on Btrfs. The whole file system on a zram device is about 260 lines using 'btrfs inspect-internal dump-tree' command. I'll excerpt just the file extents.

1. 'dd if=/dev/zero of=/mnt/test/bunchofzeros.txt bs=M count=1' results in this:

    item 7 key (257 EXTENT_DATA 0) itemoff 3409 itemsize 53
    generation 7 type 1 (regular)
    extent data disk byte 5283840 nr 1048576
    extent data offset 0 nr 1048576 ram 1048576
    extent compression 0 (none)
2. Using vi to change one character and save as the same file, ':wq', I get:

    item 7 key (262 EXTENT_DATA 0) itemoff 3409 itemsize 53
    generation 9 type 1 (regular)
    extent data disk byte 7401472 nr 1052672
    extent data offset 0 nr 1052672 ram 1052672
    extent compression 0 (none)
So you can see it's a new file, new inode, and looks like length is changed too, maybe something to do with encoding.

3. Delete that file and create a new one with the same 'dd' command as in 1.

    item 7 key (266 EXTENT_DATA 0) itemoff 3409 itemsize 53
    generation 11 type 1 (regular)
    extent data disk byte 5283840 nr 1048576
    extent data offset 0 nr 1048576 ram 1048576
    extent compression 0 (none)

    [root@flap ~]# dd conv=notrunc if=/dev/urandom bs=1 seek=524290 count=1 of=/mnt/test/bunchofzeros.txt 
    [root@flap ~]# hexdump -C /mnt/test/bunchofzeros.txt | more
    00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
    *
    00080000  00 00 5d 00 00 00 00 00  00 00 00 00 00 00 00 00  |..].............|
    00080010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
    *
    00100000

    item 7 key (266 EXTENT_DATA 0) itemoff 3409 itemsize 53
    generation 11 type 1 (regular)
    extent data disk byte 5283840 nr 1048576
    extent data offset 0 nr 524288 ram 1048576
    extent compression 0 (none)
    item 8 key (266 EXTENT_DATA 524288) itemoff 3356 itemsize 53
    generation 14 type 1 (regular)
    extent data disk byte 5242880 nr 4096
    extent data offset 0 nr 4096 ram 4096
    extent compression 0 (none)
    item 9 key (266 EXTENT_DATA 528384) itemoff 3303 itemsize 53
    generation 11 type 1 (regular)
    extent data disk byte 5283840 nr 1048576
    extent data offset 528384 nr 520192 ram 1048576
    extent compression 0 (none)
Clever. item7 and item9 actually reference the same original 1MiB data extent, but use offsets to point to the unchanged portion "creating" a 4KiB hole, and that hole is filed by item 8, a 4KiB extent containing the change. So it really did only COW one block.
Re:What’s New in Apple Filesystems [pdf] by Wowfunhappy on Tuesday June 11, 2019 @06:13PM (#20159993)
Sorry, I can’t tell, are you still talking about APFS?
Re:What’s New in Apple Filesystems [pdf] by cmurf on Tuesday June 11, 2019 @10:05PM (#20161148)
No the example under Edit 2 is Btrfs. On the details I'd expect APFS to differ. But the gist is that how changes manifest on disk do also depend on the application being used. On any file system, an edit in vi means a whole new file is written out, so it's effectively COW at an application level. Whereas an edit with an application that does byte level overwrites, would get translated into COW on a COW file system like Btrfs, APFS or ZFS. And I'd expect this COW operation to operate at the file system's minimum block level size.
Re:What’s New in Apple Filesystems [pdf] by mindajar on Tuesday June 11, 2019 @04:18PM (#20159149)
Just a block
Re:What’s New in Apple Filesystems [pdf] by duskwuff on Tuesday June 11, 2019 @02:11PM (#20157978)
Or even worse: "You no longer have any free space on your hard disk because you made a change to a large file that had a deduplicated copy."
Re:What’s New in Apple Filesystems [pdf] by voxadam on Tuesday June 11, 2019 @02:15PM (#20158022)
Block level deduplication should limit the impact of this case.
Re:What’s New in Apple Filesystems [pdf] by duskwuff on Tuesday June 11, 2019 @03:41PM (#20158841)
Only if the change is localized to specific blocks.

Besides, block-level dedup is expensive to maintain. Unless you're expecting to find a lot of duplicates -- which probably isn't the case on most end-user devices -- it's a net loss.

Re:What’s New in Apple Filesystems [pdf] by avani on Tuesday June 11, 2019 @04:33PM (#20159262)
Expect about 30%, most of it in smaller system files and libraries. Block-level dedup is great for pretty much anything where you have a consistent workflow.
Re:What’s New in Apple Filesystems [pdf] by macintux on Tuesday June 11, 2019 @08:49PM (#20160826)
A consistent workflow sounds like something you’re going to find more regularly on a server than a mobile device or home computer.
Re:What’s New in Apple Filesystems [pdf] by dictum on Tuesday June 11, 2019 @02:06PM (#20157926)
In my opinion storing, not deleting, is the most important job of a file system. Increased efficiency and free space is a better proposition than replicating the average user's mental model of free space in the volume.

(If my opinion holds true, then deduplicating is a problem when it comes to storage: a copy made to prevent bitrot/other damage to the original data doesn't work as such.)

Re:What’s New in Apple Filesystems [pdf] by Wowfunhappy on Tuesday June 11, 2019 @04:10PM (#20159084)
> Increased efficiency and free space is a better proposition than replicating the average user's mental model of free space in the volume.

I'm not sure if I agree.

In an ideal world, it makes sense that performance should be paramount. In reality, I think it's often more important that computers do what we expect them to, so we can predict when they'll break and know how to resolve the problem when they do. This is all the more important in consumer software.

"My hard drive is out of space" is a very basic computer problem that most users will run into at some point. The intuitive solution is to delete stuff. If that solution doesn't work, it could be a major problem.

Re:What’s New in Apple Filesystems [pdf] by voxadam on Tuesday June 11, 2019 @02:18PM (#20158050)
Storing vs deleting is one of the reasons that I'd love to see more research done on log based filesystems.
Re:What’s New in Apple Filesystems [pdf] by jandrese on Tuesday June 11, 2019 @02:52PM (#20158375)
It's cheaper than the memory upgrade you need to support dedup properly.
Re:What’s New in Apple Filesystems [pdf] by leiroigh on Tuesday June 11, 2019 @03:19PM (#20158623)
Not really. ZFS dedup is just brain-dead stupid. Rabin-Karp chunking plus spatial locality / opportunistic dedup allow you to gracefully scale down: you keep a complete table of successfully deduped blocks, plus a small fraction of other blocks, like e.g. smallest 1% of hashes, first blocks in files and final blocks in files. If you have a 12 MB (100 blocks) stretch that matches, you have 63% (1-1/e) chance of discovering one of these blocks as a match, and then use spatial locality to dedup adjacent blocks (duplicate blocks tend to come in bunches).
Re:What’s New in Apple Filesystems [pdf] by jandrese on Tuesday June 11, 2019 @04:56PM (#20159465)
Then why does turning on dedup on ZFS suddenly make it chomp through 4GB of memory? Is there some tuneable you're supposed to be setting to make it less insane?
Re:What’s New in Apple Filesystems [pdf] by bluedino on Tuesday June 11, 2019 @01:58PM (#20157835)
They do RAM compression, I thought disk compression was soon to follow after Apple went all-flash storage
Re:What’s New in Apple Filesystems [pdf] by yakkers on Tuesday June 11, 2019 @11:24PM (#20161513)
APFS does indeed do compression, the rationale being that compute is cheaper than I/O.

Even if most files on a typical personal computer are already inherently compressed (JPEG photos, lossy audio, etc.), you're not losing any performance, and are gaining some extra space where possible.

Re:What’s New in Apple Filesystems [pdf] by twoodfin on Tuesday June 11, 2019 @07:32PM (#20160466)
Presumably not particularly valuable since most of what takes up consumer storage these days (pictures, videos, app assets) is already in some highly compressed format.
Re:What’s New in Apple Filesystems [pdf] by ghaff on Tuesday June 11, 2019 @01:57PM (#20157816)
Relative to what? I've been around long enough it's hard for me to see just about any disk storage as expensive even if it costs more than whatever is on sale at Newegg.
Re:What’s New in Apple Filesystems [pdf] by FPGAhacker on Tuesday June 11, 2019 @03:44PM (#20158866)
Mac laptops aren’t upgradable without goofy adapters to convert nvme ssd m.2 to apple’s form factor, at which point your case won’t close up without flexing the aluminum or main board or both. For many MacBook Pro users, upgrading storage is buying a new laptop.

That’s expensive in my opinion.

OWC has made drives that fit Apple’s form factor (assuming your laptop is old enough) for a nice premium. Depending on which laptop you have, this can still be a science experiment.

Re:What’s New in Apple Filesystems [pdf] by api on Tuesday June 11, 2019 @10:21PM (#20161213)
Same with transparent compression in most cases.