Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by masklinn on Friday May 12, 2023 @08:47AM
(#35915440)
With all due respect to Carmack he’s using bzip in 2023, that’s pretty outdated on every front.
[ Parent
| on HN ]
Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by dgacmu on Friday May 12, 2023 @09:14AM
(#35915764)
You'd be surprised. There are some workloads - for me, it's geospatial data - where bzip2 clobbers all of the alternatives.
Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by krzyk on Friday May 12, 2023 @09:46AM
(#35916199)
Oh, that's interesting, I stopped using bzip2 at the time kernel sources started shipping in xz.
Do you know if there are any tests showing which compressor is better (compression wise) for which data?
Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by Aachen on Friday May 12, 2023 @09:47AM
(#35916213)
I've also noticed for some text documents (was it json? I don't remember) that bzip compresses significantly better than xz (and of course gzip/pigz). Not sure if I tested zstd with high/extreme settings at that time.
Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by elteto on Friday May 12, 2023 @10:06AM
(#35916544)
For some reason, bzip compresses text incredibly well. And it has for years, I remember noticing this almost 20 years ago.
Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by tehjoker on Friday May 12, 2023 @12:01PM
(#35918060)
It uses the Burrows-Wheeler transform to place similar strings next to each other before using other compression tricks, so it usually does a bit better.
Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by PhilipRoman on Friday May 12, 2023 @12:00PM
(#35918056)
I'm using bzip2 to compress a specific type of backup. In my case I cannot afford to steal CPU time from other running processes, so the backup process runs with severely limited CPU percentage. By crude testing I found that bzip2 used 10x less memory and finished several times faster than xz, while being very close on the compression rate.
Other algorithms like zstd and gz resulted in much lower compression rates.
I'm sure there is a more efficient solution, but changing three letters in a script was pretty much the maximum amount of effort I was going to put in.
On an unrelated note, has someone already made a meta-compression algorithm which simply picks the best performing compression algorithm for each input?
Re:Pigz: Parallel gzip for modern multi-processor, multi-cor...
by sudobash1 on Friday May 12, 2023 @01:22PM
(#35919154)
I've not seen one that picks the best compression algorithm, but I've seen ones that perform a test to try and determine if it is worth compressing. For example borg-backup software can be configured to try a light & fast compression algorithm on each chunk of data. If the chunk is compressed, then it uses a more computationally expensive algorithm to really squash it down.