Re:Show HN: We scaled Git to support 1 TB repos
by mattpallissard on Tuesday December 13, 2022 @12:38PM
(#33972474)
Yeah, which is great for storage but doesn't help over the wire.
Re:Show HN: We scaled Git to support 1 TB repos
by xmodem on Tuesday December 13, 2022 @01:05PM
(#33973029)
ZFS at least supports sending a deduplicated stream.
Re:Show HN: We scaled Git to support 1 TB repos
by mattpallissard on Tuesday December 13, 2022 @01:52PM
(#33973859)
Right, and btrfs can send a compressed stream as well, but we aren't sending raw filesystem data via VCS.
Re:Show HN: We scaled Git to support 1 TB repos
by cma on Tuesday December 13, 2022 @01:26PM
(#33973421)
One problem is if you need to support Windows clients. Microsoft charges $1600 for deduplication support or something like that: https://learn.microsoft.com/en-us/windows-server/storage/dat...
Re:Show HN: We scaled Git to support 1 TB repos
by tatersolid on Wednesday December 14, 2022 @07:49AM
(#33982762)
Deduplication is included with every version and edition of windows server since 2012.
You need to license windows server properly of course, but there is no add-on cost for deduplication.
It's always a tradeoff. Deduplication is a CPU-heavy process, and if it's done inline, it is also memory-heavy, so you're basically trading CPU and memory for storage space. It heavily depends on the use-case (and the particular FS / deduplication implementation) whether it's worth it or not
[1]: https://btrfs.wiki.kernel.org/index.php/Deduplication
[2]: https://docs.oracle.com/cd/E36784_01/html/E39134/fsdedup-1.h...