News for hackers. Stuff that matters.
Re:What’s New in Apple Filesystems [pdf] by jandrese on Tuesday June 11, 2019 @02:52PM (#20158375)
It's cheaper than the memory upgrade you need to support dedup properly.
Re:What’s New in Apple Filesystems [pdf] by leiroigh on Tuesday June 11, 2019 @03:19PM (#20158623)
Not really. ZFS dedup is just brain-dead stupid. Rabin-Karp chunking plus spatial locality / opportunistic dedup allow you to gracefully scale down: you keep a complete table of successfully deduped blocks, plus a small fraction of other blocks, like e.g. smallest 1% of hashes, first blocks in files and final blocks in files. If you have a 12 MB (100 blocks) stretch that matches, you have 63% (1-1/e) chance of discovering one of these blocks as a match, and then use spatial locality to dedup adjacent blocks (duplicate blocks tend to come in bunches).
Re:What’s New in Apple Filesystems [pdf] by jandrese on Tuesday June 11, 2019 @04:56PM (#20159465)
Then why does turning on dedup on ZFS suddenly make it chomp through 4GB of memory? Is there some tuneable you're supposed to be setting to make it less insane?