Pigz: Parallel gzip for modern multi-processor, multi-core machines

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by dale_glass on Friday May 12, 2023 @08:48AM (#35915443)

Unix generally favors processes over threads, at least the old school Unix. Threads are a more recent innovation.

The old approach was that programs don't need internal parallelism because you can get it by just piping stuff and relying on the kernel's buffering to keep multiple processes busy.

Eg, tar is running on one core dealing with the filesystem, gzip is running on another core compressing stuff.

In the early days, Windows would have a single program doing everything (eg, Winzip) on a single core, while Unix would have a pipeline with the task split into multiple programs that would allow for this implicit parallelism and perform noticeably better.

Today this is all old and doesn't cut it anymore on 128 core setups.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by saagarjha on Friday May 12, 2023 @09:13AM (#35915752)

Well, if you split the file into chunks you could fan it across cores by compressing each individually.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by frde on Friday May 12, 2023 @10:03AM (#35916501)

Having to do things like this is exactly the problem.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by loeg on Friday May 12, 2023 @10:04AM (#35916512)

“Recent” as in the 80s or 90s, sure. Threads are older than unix was when threads were introduced.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by fnordpiglet on Friday May 12, 2023 @11:42AM (#35917788)

I’m not sure I understand. Typically I’ve thought of threads called “threads” being formalized with POSIX threads. Before that, while my memory is vague on this, it was just lumped into multiprocessing under various names.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by loeg on Friday May 12, 2023 @04:03PM (#35921572)

Even if you ignore threading models prior to pthreads, pthreads itself dates to 1995.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by fnordpiglet on Friday May 12, 2023 @04:22PM (#35921845)

And Unix 1973

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by loeg on Friday May 12, 2023 @06:51PM (#35923544)

So: Unix was ~22 when Pthreads were introduced in 1995, and Pthreads are 28 years old now.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by wanttocomment on Friday May 12, 2023 @10:30AM (#35916890)

I agree with most of these points except blaming this on processes vs threads. The only difference is all memory being shared by default, vs explicitly deciding what memory to share. With all the emphasis on memory safety on HN you think this point would be appreciated.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by fnordpiglet on Friday May 12, 2023 @11:40AM (#35917765)

That’s fairly new, with threads and processes becoming basically the same. Historically threads didn’t exist, then they were horrifically implemented and non standard, then they standardized and were horrifically implemented, then they were better implemented but the APIs were difficult to use safely, etc etc. Also threads were much more light weight than a process. This shifted with light weight processes, etc.

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by wanttocomment on Friday May 12, 2023 @02:48PM (#35920505)

That's true, but if you're looking that far back, multicore is new too

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by fnordpiglet on Friday May 12, 2023 @04:23PM (#35921856)

I guess “that far back” becomes different as you get older :-) it doesn’t seem that long ago to me :-)

[ Parent | on HN ]

Re:Pigz: Parallel gzip for modern multi-processor, multi-cor... by mhh__ on Friday May 12, 2023 @07:13PM (#35923697)

Processes can still be fine. If you're lucky you can be debating over whether you'd like something to be 99x or 100x faster.

[ Parent | on HN ]