News for hackers. Stuff that matters.

A comment on Deduplicating a 10.4 TiB game preservation archive (WIP). Parent comment

Re:Deduplicating a 10.4 TiB game preservation archive (WIP) by Intralexical on Wednesday December 18, 2024 @03:49AM (#42449016)
This is good thinking, but I think you are basically describing a Restic/Borg respository :)

- Deduplication? Check.

- Compact format? Check.

- Good read speed? Yep. (Proportional to backing store.)

- Custom purpose-built? Yeah, that's what backup programs are for.

- Custom data layout? Check. (Rabin/BuzHash content-defined chunking, SHA256 dedupe.)

- Adding additional versions? Yes. ("Incremental backups"— See above.)

- Virtual filesystem access? `restic mount`/`borg mount`.

[ Parent | on HN ]
Re:Deduplicating a 10.4 TiB game preservation archive (WIP) by DrFrugal on Wednesday December 18, 2024 @05:45AM (#42449478)
thanks for mentioning those 2 projects, will check them out over the holidays and do some experimenting ^^
[ Parent | on HN ]
Re:Deduplicating a 10.4 TiB game preservation archive (WIP) by mappu on Wednesday December 18, 2024 @02:33PM (#42454039)
The same storage system is used by some non-backup software too, Perkeep and Seafile
[ Parent | on HN ]