News for hackers. Stuff that matters.
Re:For the Love of Pipes by crazygringo on Tuesday January 22, 2019 @12:40PM (#18970284)
Pipes are awesome and infuriating.

Sometimes they work great -- being able to dump from MySQL into gzip sending across the wire via ssh into gunzip and into my local MySQL without ever touching a file feels nothing short of magic... although the command/incantation to do so took quite a while to finally get right.

But far too often they inexplicably fail. For example, I had an issue last year where piping curl to bunzip would just inexplicably stop after about 1GB, but it was at a different exact spot every time (between 1GB and 1.5GB). No error message, no exit, my network connection is fine, just an infinite timeout. (While curl by itself worked flawlessly every time.)

And I've got another 10 stories like this (I do a lot of data processing). Any given combination of pipe tools, there's a kind of random chance they'll actually work in the end or not. And even more frustrating, they'll often work on your local machine but not on your server, or vice-versa. And I'm just running basic commodity macOS locally and out-of-the-box Ubuntu on my servers.

I don't know why, but many times I've had to rewrite a piped command as streams in a Python script to get it to work reliably.

Re:For the Love of Pipes by invsblduck on Tuesday January 22, 2019 @02:16PM (#18971371)
> Any given combination of pipe tools, there's a kind of random chance they'll actually work in the end or not.

While this may be your experience, the mechanism of FIFO pipes in Unix (which is filehandles and buffers, basically), is an old one that is both elegant and robust; it doesn't "randomly" fail due to unreliability of the core algorithm or components. In 20 years, I never had an init script or bash command fail due to the pipe(3) call itself being unreliable.

If you misunderstand detailed behavior of the commands you are stitching together--or details of how you're transiting the network in case of an ssh remote command--then yes, things may go wrong. Especially if you are creating Hail Mary one-liners, which become unwieldy.

Re:For the Love of Pipes by laumars on Tuesday January 22, 2019 @02:34PM (#18971581)
If got to agree. I can’t recall a pipe ever failing due to unreliability.

One issue I did used to have (before I discovered ‘-o pipefail’[1]) was the annoyance that if an earlier command in a pipeline failed, all the other commands in the pipeline still ran albeit with no data or garbage data being piped to them.

[1] https://stackoverflow.com/questions/1550933/catching-error-c...

Re:For the Love of Pipes by EE84M3i on Tuesday January 22, 2019 @01:32PM (#18970871)
Perhaps your example was contrived, but why would you pipe into gzip instead of using transparent ssh compression?
Re:For the Love of Pipes by crazygringo on Tuesday January 22, 2019 @01:54PM (#18971132)
Because it simply never occurred to me to check if ssh would have compression built-in.

Because why would it? If the UNIX philosophy is to separate out tools and pipe them, then the UNIX philosophy should be to pipe through gzip and gunzip, not for ssh to provide its own redundant compression option, right?

Re:For the Love of Pipes by acdha on Tuesday January 22, 2019 @02:04PM (#18971239)
This is a good example of where that simple rule breaks down: piping it would only work when you are running a command and feeding its output to a different location whereas having it in SSH helps with everything so e.g. typing `ls` in a big directory, `cat`-ing a file, using `scp` on text, etc. benefits.
Re:For the Love of Pipes by oweiler on Tuesday January 22, 2019 @12:44PM (#18970331)
I've built a pipe which in very rare cases ran into a segfault. Never found out why.