pigz: A parallel implementation of gzip for multi-core machines

Re:pigz: A parallel implementation of gzip for multi-core ma... by mgerdts on Monday October 17, 2022 @08:19PM (#33241399)

pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation.

A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code churn that makes it a pain to apply to the current version.

[ Parent | on HN ]

Re:pigz: A parallel implementation of gzip for multi-core ma... by fulafel on Tuesday October 18, 2022 @01:51AM (#33243438)

Docker is indeed looking for a "unpigz" executable to use: https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27...

So interesting if they implemented and tested that and get only a marginal CRC speedup.

edit: someone here seems to observe a ~ doubling with unpigz vs zcat: https://unix.stackexchange.com/a/363739

[ Parent | on HN ]

Re:pigz: A parallel implementation of gzip for multi-core ma... by mxmlnkn on Tuesday October 18, 2022 @02:49AM (#33243838)

igzip is even faster. See also my answer that contains a quick benchmark in that linked question at the bottom.

[ Parent | on HN ]

Re:pigz: A parallel implementation of gzip for multi-core ma... by mxmlnkn on Tuesday October 18, 2022 @02:25AM (#33243648)

I recently implemented pragzip for parallel gzip decompression https://github.com/mxmlnkn/pragzip . I would be interested to know how your PR back then worked to parallelize the decompression.

[ Parent | on HN ]