Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by rhinoceraptor on Friday May 12, 2023 @10:44AM
(#35917051)
Zstandard has a dictionary functionality which allows you to pre-train on sample data to achieve higher compression ratios and faster compression on large numbers of small files.
[ Parent
| on HN ]
Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by cout on Friday May 12, 2023 @10:57AM
(#35917228)
Interesting - does it fit that automatically or is there a manual step?
Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by sgtnoodle on Friday May 12, 2023 @11:52AM
(#35917931)
Don't you then need to store that dictionary somewhere out of band? It seems like you would still need a tar-like format to manage that, at which point as an archive format it seems more complicated with a worse separation of concerns.