News for hackers. Stuff that matters.
Re:Unix: When pipes get names by telephonetemp on Tuesday October 08, 2013 @07:01PM (#6518421)
Are there any particularly clever uses for named pipes in shell scripts? I don't think I've ever seen any.
Re:Unix: When pipes get names by ars on Tuesday October 08, 2013 @07:14PM (#6518488)
Example 1:

In one shell:

mplayer -input file=named_pipe

In a different shell:

echo 'pause' > named_pipe

or:

echo 'quit' > named_pipe

So you can control your music player easily. (I bind the echo script to hotkeys.)

Example 2:

mysql LOAD DATA INFILE wants a filename. So I made a named pipe and sent my data there (putting that command in the background since it will hang till the data is read). Then I use that named pipe as the filename for load data.

Example 3:

You want to send your data over ssh, but you also want to send commands over ssh (you can't do both at the same time, except in the command line which might not be enough). You could send the data over a second ssh session - but then you can't do your preliminary commands first.

So you make a named pipe on the server and send the data to it in one ssh session. Then you open a second ssh session, do your preliminary commands, then your data command using the named pipe as a filename.

(There are other ways to do this with renaming file descriptors in a shell, but a named pipe is easier, and much more flexible.)

Example 4:

You want to start a long running process that waits for data. You let it read from the named pipe, and then you can write to the named pipe from any other program as needed.

Re:Unix: When pipes get names by gwern on Tuesday October 08, 2013 @09:45PM (#6518997)
I actually did exactly this exactly a few years ago (I thought I was so clever). I had a video with hardcoded subtitles, which I wanted to transcribe; I couldn't just let mplayer run because I cannot transcribe in realtime, and slowing it down would still be a problem because it would take much more time and would necessitate backtracking.

So, I set up a FIFO like described, and then I bound the key '1' in Emacs to echo 'p' to it. Voila. I could seamlessly pause and unpause the video while typing away furiously. The strategy was quite a hack, but it worked very well for my purpose.

Re:Unix: When pipes get names by Hello71 on Tuesday October 08, 2013 @08:39PM (#6518809)
> You want to send your data over ssh, but you also want to send commands over ssh (you can't do both at the same time, except in the command line which might not be enough). You could send the data over a second ssh session - but then you can't do your preliminary commands first.

SSH already has a Control facility for this.

Re:Unix: When pipes get names by ars on Tuesday October 08, 2013 @08:44PM (#6518831)
That won't help. How would you feed the data to stdin of a command run in a different process?

The Control facility is for multiplexing the data streams, but that solves a different problem.

Re:Unix: When pipes get names by nine_k on Tuesday October 08, 2013 @09:47PM (#6519001)
The Control facility lets you reuse the connection, bit not the session. So you still need two sessions, say, one ssh and one scp.
Re:Unix: When pipes get names by gizmo686 on Tuesday October 08, 2013 @07:47PM (#6518623)
I don't think named pipes work particuarly well in shell scripts. A program writing to a pipe will not terminate until the output is consumed. However, consuming the output will not happen until the program finishes and the script arrives that the program that reads from the pipe. Similarly, a program reading from the pipe will not terminate until the program writing to it does. If you start reading before you start writing, then you simply wait. Anonymous pipes solves this problem by launching all of the programs at the same time. You could work around this by telling the shell to run the command in the background (append & in bash).

Having said that, I have had occasion to use named pipes in a shell script. Basically, I had two programs and I wanted to determine which lines were in one, but not both, of their outputs. Using named pipes, I was able to do something like:

  mkfifo foo
  ./prog1 > foo &
  ./prog2 > foo &
  cat foo | sort | uniq -u
Effectively, this provides a way to combine the output of multiple commands. I took it as a matter of faith that doing this won't interleave lines. I assume that as long as both programs flush only at line breaks then there is not any problem.
Re:Unix: When pipes get names by bcoates on Tuesday October 08, 2013 @09:03PM (#6518891)
Agreed, every time I've used a named pipe in a shellscript it's caused me nothing but pain.

IIRC, they're guaranteed not to interleave write()s less than PIPE_BUF (4k?) but if your program is buffering internally it might not end writes on a newline.

Re:Unix: When pipes get names by dllthomas on Tuesday October 08, 2013 @08:16PM (#6518741)
You don't need named pipes for this:

{ ./prog1; ./prog2; } | sort | uniq -u

Re:Unix: When pipes get names by derleth on Tuesday October 08, 2013 @08:11PM (#6518717)
Does:

    sort < foo | uniq -u
not work with named pipes?
Re:Unix: When pipes get names by dllthomas on Tuesday October 08, 2013 @08:18PM (#6518749)
It works fine.

The advantage to

    cat foo | sort | uniq -u
is that it is (marginally) easier to drop in another filter in front of sort.
Re:Unix: When pipes get names by knz42 on Wednesday October 09, 2013 @02:52AM (#6519776)
This argument is less strong in the face of:

  <foo  sort | uniq -u
(redirections can be at the beginning too.)

Also: sort -u instead of sort|uniq.

Re:Unix: When pipes get names by dllthomas on Wednesday October 09, 2013 @11:04AM (#6521651)
It's still a reason to disprefer the form that was given. I don't see any reason to prefer the cat version over yours, except habits, though. That said, there's only weak reason to avoid the cat.
Re:Unix: When pipes get names by jl6 on Wednesday October 09, 2013 @12:42PM (#6522307)
It is also simpler, in the sense that it presents a left-to-right order that visually illustrates the pipeline, without the incongruous left-facing angle bracket.
Re:Unix: When pipes get names by sikhnerd on Tuesday October 08, 2013 @07:10PM (#6518474)
Only time I've used it is as a poor man's screen -x

  mkfifo file; script -f file
Other user simply does

  cat file
Re:Unix: When pipes get names by smlacy on Tuesday October 08, 2013 @07:49PM (#6518633)
Sort a file in "parallel" (note quotes!) in a multicore machine (note I'm showing 2 cores but this could be generalized to more):

    mkfifo split_aa
    mkfifo split_ab
    mkfifo sorted_aa
    mkfifo sorted_ab
    split -n2 input_file split_ & 
    sort split_aa >> sorted_aa & 
    sort split_ab >> sorted_ab & 
    sort -m sorted_aa sorted_bb > final_sorted
It's left as an exercise for the reader to determine if this actually saves any time, and if this technique could be used to implement a truly parallel sort.
Re:Unix: When pipes get names by timClicks on Wednesday October 09, 2013 @01:43AM (#6519608)
What about?

    sort --parallel=2 input_file
Re:Unix: When pipes get names by smlacy on Wednesday October 09, 2013 @02:31PM (#6522941)
Well, yeah, that's the easy way. :)

This is mostly to illustrate a technique of "parallel processing using fifos and split input" It's almost like a miniature mapreduce running locally. See also "bashreduce" (https://github.com/erikfrey/bashreduce) which I didn't write.

Re:Unix: When pipes get names by cynwoody on Tuesday October 08, 2013 @07:44PM (#6518610)
In this discussion† from yesterday, the task of finding the longest path below a directory came up. You can see both a clumsy solution, using a named pipe and iterative Python, and an elegant one-liner (in a reply), using an anonymous pipe and functional Python.

This is not to say named pipes do not have their uses. Just that you should think before reaching for them first.

†https://news.ycombinator.com/item?id=6510640

Re:Unix: When pipes get names by unhammer on Wednesday October 09, 2013 @10:14AM (#6521331)
If you have a slow-starting program, you can use named pipes as its stdin/stdout. Then write a little script that writes to the inpipe and reads from the outpipe (interleavedly, so as not to fill up the pipe buffer, I think that's about 4k or so). Now that little script starts up "instantly" compared to the original slow-starting program. http://mywiki.wooledge.org/NamedPipes shows an outline of it.

http://mywiki.wooledge.org/ProcessManagement#I_want_to_proce.... is another neat trick: when a background process ends, make it echo to some pipe, some other "controller" process is continually reading from that pipe. Since the read is blocking, it'll wait patiently for the echo until it does whatever it should do (maybe even based on whatever it read from the pipe). So much less hacky than sleep-polling :-)

Re:Unix: When pipes get names by smartaleckkill on Tuesday October 08, 2013 @07:25PM (#6518536)
I use a FIFO to pipe alpine's new mail alert to notifyd; not sure how clever that is, but it works for me.
Re:Unix: When pipes get names by ars on Tuesday October 08, 2013 @08:34PM (#6518794)
Can you post the script? I'd love to use it.
Re:Unix: When pipes get names by smartaleckkill on Wednesday October 09, 2013 @07:50AM (#6520656)
Used to use my own wee C prog that just passed the alert unparsed to libnotify; currently using a modified version of this far superior shell script (which I can't take any credit for): https://github.com/Bruce-Connor/alpine-osd-notify/blob/maste... (You have to set newmail-fifo-path in your pinerc.)
Re:Unix: When pipes get names by christianmann on Wednesday October 09, 2013 @12:22AM (#6519374)
Poor man's (unauthenticated!) remote shell:

    mkfifo /tmp/buf;
    cat /tmp/buf | /bin/sh -i 2>&1 | nc -lp 31337 > /tmp/buf
And then, in another terminal:

    nc 127.0.0.1 31337