News for hackers. Stuff that matters.
Re:Show HN: We scaled Git to support 1 TB repos by synergy20 on Tuesday December 13, 2022 @11:59AM (#33971654)
can you be more specific here,very interested
Re:Show HN: We scaled Git to support 1 TB repos by dark-star on Tuesday December 13, 2022 @12:17PM (#33972001)
There are filesystems that support inline or post-process deduplication. btrfs[1] and zfs[2] come to mind as free ones, but there are also commercial ones like WAFL etc.

It's always a tradeoff. Deduplication is a CPU-heavy process, and if it's done inline, it is also memory-heavy, so you're basically trading CPU and memory for storage space. It heavily depends on the use-case (and the particular FS / deduplication implementation) whether it's worth it or not

[1]: https://btrfs.wiki.kernel.org/index.php/Deduplication

[2]: https://docs.oracle.com/cd/E36784_01/html/E39134/fsdedup-1.h...

Re:Show HN: We scaled Git to support 1 TB repos by mattpallissard on Tuesday December 13, 2022 @12:38PM (#33972474)
Yeah, which is great for storage but doesn't help over the wire.
Re:Show HN: We scaled Git to support 1 TB repos by xmodem on Tuesday December 13, 2022 @01:05PM (#33973029)
ZFS at least supports sending a deduplicated stream.
Re:Show HN: We scaled Git to support 1 TB repos by mattpallissard on Tuesday December 13, 2022 @01:52PM (#33973859)
Right, and btrfs can send a compressed stream as well, but we aren't sending raw filesystem data via VCS.
Re:Show HN: We scaled Git to support 1 TB repos by cma on Tuesday December 13, 2022 @01:26PM (#33973421)
One problem is if you need to support Windows clients. Microsoft charges $1600 for deduplication support or something like that: https://learn.microsoft.com/en-us/windows-server/storage/dat...
Re:Show HN: We scaled Git to support 1 TB repos by tatersolid on Wednesday December 14, 2022 @07:49AM (#33982762)
Deduplication is included with every version and edition of windows server since 2012. You need to license windows server properly of course, but there is no add-on cost for deduplication.
Re:Show HN: We scaled Git to support 1 TB repos by dark-star on Wednesday December 14, 2022 @07:43AM (#33982720)
there exists an open-source btrfs filesystem driver for Windows...
Re:Show HN: We scaled Git to support 1 TB repos by alchemist1e9 on Tuesday December 13, 2022 @01:30PM (#33973502)
zbackup is a great block level deduplication trick.