News for hackers. Stuff that matters.
Re:Pigz - multicore version of gzip. by dryicerx on Wednesday March 31, 2010 @07:49PM (#1233375)
I am just curious, why not implement this within gzip instead of forking it out as a completely new project?
Re:Pigz - multicore version of gzip. by jey on Wednesday March 31, 2010 @09:33PM (#1233557)
What's the benefit of that? I see downsides: have to deal with the FSF, have to work with the existing codebase, have to fight to get the changes accepted, etc. It can always be ported into gzip if that becomes desirable.
Re:Pigz - multicore version of gzip. by krainboltgreene on Thursday April 01, 2010 @01:29AM (#1233943)
have to fight to get the changes accepted

One of the main reasons I stay the hell away from OSS unless I can fork it.

Re:Pigz - multicore version of gzip. by Rexxar on Thursday April 01, 2010 @05:35AM (#1234227)
The FSF is not involved in the development of zlib. Moreover, pigz and zlib are maintained/developed by the same person.
Re:Pigz - multicore version of gzip. by jey on Thursday April 01, 2010 @05:04PM (#1235379)
Non-sequitur? gzip != zlib
Re:Pigz - multicore version of gzip. by Rexxar on Thursday April 01, 2010 @09:24PM (#1235798)
gzip ~= zlib, It's the same authors and the same algorithms.
Re:Pigz - multicore version of gzip. by ulrich on Thursday April 01, 2010 @06:21AM (#1234280)
That was exactly my thought when seeing the website.
Re:Pigz - multicore version of gzip. by anotherjesse on Wednesday March 31, 2010 @07:32PM (#1233349)
This is REALLY useful when you are bundling AMI images.

When we changed NASA Nebula from XEN to KVM we had to tweak a few files in every image, this sped up the conversion by minutes per image.

Re:Pigz - multicore version of gzip. by jey on Wednesday March 31, 2010 @07:37PM (#1233353)
Re:Pigz - multicore version of gzip. by lsb on Wednesday March 31, 2010 @07:39PM (#1233356)
The next parallelized compressor I'd like to see is LZMA. It's basically Lempel-Zip, like gzip, but with a much wider window (tens of megs, vs a few dozen k), and because it's dictionary-based compression it decompresses much faster than bzip.
Re:Pigz - multicore version of gzip. by wmf on Wednesday March 31, 2010 @08:34PM (#1233451)
http://jnovy.fedorapeople.org/pxz/

It's definitely needed since XZ is fairly slow.

Re:Pigz - multicore version of gzip. by MikeCapone on Wednesday March 31, 2010 @07:28PM (#1233340)
Sounds good. If it works well, I'd love to see it become the new default on all *nix distros. It's about time some of these basic tools get updated for multi-core hardware.
Re:Pigz - multicore version of gzip. by natfriedman on Wednesday March 31, 2010 @08:11PM (#1233411)
We used this in SUSE Studio to decrease build time. Worked well.

There's also MPIBZIP2 which uses (IIRC) libmpi to do bzip2 compression/decompression across several machines. We messed with that too, but it wasn't worth the added complexity.

Re:Pigz - multicore version of gzip. by Luyt on Wednesday March 31, 2010 @07:46PM (#1233371)
It's in FreeBSD's ports:

  /usr/ports/archivers/pigz            Parallel GZIP
Re:Pigz - multicore version of gzip. by _b8r0 on Thursday April 01, 2010 @02:51AM (#1234042)
Just out of interest, does anyone have any numbers on whether or not the problems with gzip are more i/o bound or cpu bound? I would've thought with gzip i/o would've been a bigger factor, but am happy to be proven wrong.
Re:Pigz - multicore version of gzip. by kvs on Wednesday March 31, 2010 @09:45PM (#1233570)
Anyone looked at the source yet? Is it just multi-threaded or actually using parallel frameworks like ITBB or OpenMP?
Re:Pigz - multicore version of gzip. by sparky on Thursday April 01, 2010 @12:57AM (#1233903)
It uses pthreads. The source is very approachable and small, if you want to dig a bit more.
Re:Pigz - multicore version of gzip. by pan69 on Wednesday March 31, 2010 @07:43PM (#1233363)
"How is it pronounced?"

Australians would pronounce it as "pig-zed". Just one of those novelties I guess... :)

Re:Pigz - multicore version of gzip. by javert on Wednesday March 31, 2010 @09:04PM (#1233504)
According to the site, it's actually pronounced "pig-zee."

(I quite dislike the name, myself. And I think everyone's actually going to pronounce it "pigs".)

Re:Pigz - multicore version of gzip. by jlangenauer on Wednesday March 31, 2010 @08:33PM (#1233447)
I'm Australian, and I'd pronounce it "pigs".
Re:Pigz - multicore version of gzip. by jsankey on Wednesday March 31, 2010 @07:53PM (#1233379)
Unless, perhaps, they have memories of watching Monkey during their childhood:

http://en.wikipedia.org/wiki/Monkey_(TV_series)

Re:Pigz - multicore version of gzip. by abstractbill on Wednesday March 31, 2010 @07:59PM (#1233391)
I would guess "pigs", as in "adding cores makes pigz fly" ;)