News for hackers. Stuff that matters.

A comment on Deduplicating a 10.4 TiB game preservation archive (WIP).

Re:Deduplicating a 10.4 TiB game preservation archive (WIP) by sillystuff on Tuesday December 17, 2024 @06:00PM (#42446379)
Instead of dupremove, bees may be more appropriate for your goals.

https://github.com/Zygo/bees

Bees does things at a lower level than just reflinking full files like dupremove (it dedups at the level of extents). Be sure to read the links in the "Recommended Reading" section, of the above link, to hopefully avoid issues.

[ Parent | on HN ]
Re:Deduplicating a 10.4 TiB game preservation archive (WIP) by DrFrugal on Tuesday December 17, 2024 @06:17PM (#42446501)
duperemove has "--dedupe-options=partial" which also enables this, not just full extents. the issue still is, that the data within the archive is not block aligned, thus preventing me from deduplicating them properly
[ Parent | on HN ]