pigz: A parallel implementation of gzip for multi-core machines

Re:pigz: A parallel implementation of gzip for multi-core ma... by xfalcox on Monday October 17, 2022 @05:24PM (#33239687)

One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a nice speedup for us, since we use large container images and automatic scaling for incoming traffic surges.

[ Parent | on HN ]

Re:pigz: A parallel implementation of gzip for multi-core ma... by danuker on Monday October 17, 2022 @05:26PM (#33239712)

Have you optimized the low-hanging fruit in your image size?

Because compression programs are as high-hanging fruit as you can get, and parallelizing them can only be done once.

[ Parent | on HN ]

Re:pigz: A parallel implementation of gzip for multi-core ma... by chasil on Monday October 17, 2022 @06:15PM (#33240208)

I think dracut also uses pigz to create the initrd when installing a new Linux kernel rpm package.

[ Parent | on HN ]

Re:pigz: A parallel implementation of gzip for multi-core ma... by anistas on Tuesday October 18, 2022 @10:41AM (#33247941)

Why do you need a heavy multi-thread compressor if modern initramfs systems (like https://github.com/anatol/booster) create small image of size 2MiB and below?

You won't see any improvement from parallelization on this type of data.

[ Parent | on HN ]

Re:pigz: A parallel implementation of gzip for multi-core ma... by mgerdts on Monday October 17, 2022 @08:19PM (#33241399)

pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation.

A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code churn that makes it a pain to apply to the current version.

[ Parent | on HN ]

Re:pigz: A parallel implementation of gzip for multi-core ma... by fulafel on Tuesday October 18, 2022 @01:51AM (#33243438)

Docker is indeed looking for a "unpigz" executable to use: https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27...

So interesting if they implemented and tested that and get only a marginal CRC speedup.

edit: someone here seems to observe a ~ doubling with unpigz vs zcat: https://unix.stackexchange.com/a/363739

[ Parent | on HN ]

Re:pigz: A parallel implementation of gzip for multi-core ma... by mxmlnkn on Tuesday October 18, 2022 @02:49AM (#33243838)

igzip is even faster. See also my answer that contains a quick benchmark in that linked question at the bottom.

[ Parent | on HN ]

Re:pigz: A parallel implementation of gzip for multi-core ma... by mxmlnkn on Tuesday October 18, 2022 @02:25AM (#33243648)

I recently implemented pragzip for parallel gzip decompression https://github.com/mxmlnkn/pragzip . I would be interested to know how your PR back then worked to parallelize the decompression.

[ Parent | on HN ]