Pigz: Parallel gzip for modern multi-processor, multi-core machines

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by cout on Friday May 12, 2023 @08:41AM (#35915379)

For many small files, compressing first will compress worse, because each file has its own dictionary; compressing last means you can take advantage of similarities in files to improve compression ratio.

Compressing first can also be slower if the average file size is smaller than the block size, because the main thread cannot queue new jobs as fast as cores complete them (this happens e.g. with 7zip at fast compression settings with solid archive turned off). Tarring then compressing means small files can be aggregated into a single block, giving both good speed and compression ratio.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by rhinoceraptor on Friday May 12, 2023 @10:44AM (#35917051)

Zstandard has a dictionary functionality which allows you to pre-train on sample data to achieve higher compression ratios and faster compression on large numbers of small files.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by cout on Friday May 12, 2023 @10:57AM (#35917228)

Interesting - does it fit that automatically or is there a manual step?

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by sgtnoodle on Friday May 12, 2023 @11:52AM (#35917931)

Don't you then need to store that dictionary somewhere out of band? It seems like you would still need a tar-like format to manage that, at which point as an archive format it seems more complicated with a worse separation of concerns.

[ Parent | on HN ]