Re:Bash process substitution
by skywhopper on Sunday February 08, 2015 @07:24PM
(#9019175)
I use this a lot in a couple of different contexts.
First, it's a convenient way to turn around pipelines inside scripts in situations where you need to feed a command output or pipeline into a read loop but the loop needs to be part of the original script process so it can manipulate variables, etc. Meaningless example of optionally incrementing a counter in a loop based on the output of a pipeline of commands:
while read LINE; do
if grep -q xyz <<< "$LINE"; then
COUNTER=$(( $COUNTER + 1 ))
fi
done < <(awk '{print $4,$7}' infile.txt |cut -d, -f3)
If you did it in the "awk ...|cut ...| while read LINE..." order, COUNTER would be getting incremented in a subshell, and the value wouldn't stick around after the loop.
The other most common way I use it is to diff a couple of sets of pipelines or even just files that need sorting before comparison:
diff <(sort -u file1.txt) <(sort -u file2.txt)
Saves a ton of steps. As pointed out by jkbyc, the >() construction for output process substitution becomes really powerful when combined with tee.
Re:Bash process substitution
by skywhopper on Monday February 09, 2015 @07:29AM
(#9020811)
Another example of using diff on two process substitutions I did this morning. I've been quarantining some files from various paths that all have unique basenames into a single directory. I have a CSV file listing the original path in the first field followed by some other metadata. I wanted to compare the files in the CSV file ("quarantined-files.csv") to the files I had moved into the "quarantine" subdirectory. I came up with this:
(Yes, the listing from "ls" comes out sorted, but it's always possible it'll sort certain characters a bit differently than "sort" does depending on locale settings, aliases, etc, etc. Piping both outputs through "sort" should ensure the same sorting collation in all cases.)
It took me a while to really grok this sort of thing, but once it clicked, it's often the first thing I think of. Basically any time you might be tempted to dump something to a temporary file, you should ask yourself "can I use process substitution?"
Re:Bash process substitution
by jonaf on Sunday February 08, 2015 @07:53PM
(#9019253)
One neat trick I've found is using `exec` in conjunction with process substitution to effect and gather changes to the current environment within the context of a script (usually not as useful with a pty attached).
Example: how can you redirect stdout to syslog, a specified log file, and stdout? Easy, with tee, right?
```
$my_command | tee -a >(logger -t "$0[my_command]" -p "local0.INFO") "/var/log/my_logs/$(date +%s).log")
```
Okay, that's cool. But, I have to run it for every command in my script! How inconvenient. Is there a way we can make all stdout for the script do this?
Yes! Use `exec`!
```
exec &> tee -a >(logger -t "$0[my_command]" -p "local0.INFO") "/var/log/my_logs/$(date +%s).log")
Re:Bash process substitution
by barrkel on Sunday February 08, 2015 @12:33PM
(#9017812)
I use this all the time. It's at its most useful when you want to use a process as a source or sink, and avoid using a temporary file as input or output when a tool that doesn't operate on standard input or output. Often, these tools need to work with multiple files that have different semantic intent.
For example, `grep -f <pattern-file>` - grep works on standard input, but if you also want to provide patterns via process output, you need to use temporary files, fifos, or process substitution.
Re:Bash process substitution
by ptx on Sunday February 08, 2015 @01:39PM
(#9018040)
I've never understood the point of tee without process substitution. I guess you could do the same thing manually by opening another file descriptor first an forking some process, but is it of any use in a normal simple pipe?
Re:Bash process substitution
by derefr on Sunday February 08, 2015 @04:37PM
(#9018678)
That seems like an anti-unixism (operating with multiple EUIDs in a single process group), though I'm not too sure what the equivalent unixism would be—maybe something with a daemon and a spool directory? Or a user-writable FIFO, redirected to the correct file by the elevated user in a separate command?
Re:Bash process substitution
by mturmon on Sunday February 08, 2015 @03:22PM
(#9018392)
I use tee to save log files from long running programs. The chatter goes to stdout, and informs me about progress. But the tee also saves the chatter, so I can review it if something goes wrong.
You could get a similar effect in other ways (e.g., tail -f, or terminal scrollback), but I find tee is convenient.
Re:Bash process substitution
by philsnow on Sunday February 08, 2015 @06:09PM
(#9018972)
Often I'll do something like
long_running_thing | tee long_running_thing_stdout
and want to see the first few lines of output before going to do something else. The pipe causes buffering on stdout, though, so I have to sit there until 4k of output has come out on stdout.
Re:Bash process substitution
by stringy on Sunday February 08, 2015 @01:23PM
(#9017996)
The fish[1] shell also has process substitution (without requiring additional syntax) in the form of psub (which is implemented simply as a fish function; see `type psub` from a fish shell):
Re:Bash process substitution
by falcolas on Sunday February 08, 2015 @01:09PM
(#9017934)
If you would like to become comfortable with using bash process substitution, I recommend figuring out how many of the 'moreutils' utilities can be replicated using only bash process substitution and your generic linux utilities. It makes for a fun exercise.
Re:Bash process substitution
by _ZeD_ on Sunday February 08, 2015 @12:37PM
(#9017829)
Diff and diff3 support reading from /deve/fdXX ... unfortunalety I don't know a graphic differenza (meld, kdiff3, ...) supporting this kind of input. It's a shame, 'cause I frequently want to diff the output of two program (or the same program with different inputs)
Re:Bash process substitution
by kolev on Sunday February 08, 2015 @07:25PM
(#9019180)
Although I can write a lot of my stuff in (much) higher-level languages like Python or even Perl, I prefer to do it in Bash. The thing that I hate the most is the lack of basic data structures - especially as function returns.
First, it's a convenient way to turn around pipelines inside scripts in situations where you need to feed a command output or pipeline into a read loop but the loop needs to be part of the original script process so it can manipulate variables, etc. Meaningless example of optionally incrementing a counter in a loop based on the output of a pipeline of commands:
If you did it in the "awk ...|cut ...| while read LINE..." order, COUNTER would be getting incremented in a subshell, and the value wouldn't stick around after the loop.The other most common way I use it is to diff a couple of sets of pipelines or even just files that need sorting before comparison:
Saves a ton of steps. As pointed out by jkbyc, the >() construction for output process substitution becomes really powerful when combined with tee.It took me a while to really grok this sort of thing, but once it clicked, it's often the first thing I think of. Basically any time you might be tempted to dump something to a temporary file, you should ask yourself "can I use process substitution?"
Example: how can you redirect stdout to syslog, a specified log file, and stdout? Easy, with tee, right?
``` $my_command | tee -a >(logger -t "$0[my_command]" -p "local0.INFO") "/var/log/my_logs/$(date +%s).log") ```
Okay, that's cool. But, I have to run it for every command in my script! How inconvenient. Is there a way we can make all stdout for the script do this?
Yes! Use `exec`!
``` exec &> tee -a >(logger -t "$0[my_command]" -p "local0.INFO") "/var/log/my_logs/$(date +%s).log")
$my_command ```
It's not too long and very well written so it's an enjoyable read.
You almost always learn something new which you can integrate into your workflow.
http://www.gnu.org/software/bash/manual/bashref.html
For example, `grep -f <pattern-file>` - grep works on standard input, but if you also want to provide patterns via process output, you need to use temporary files, fifos, or process substitution.
Here's an example:
You could get a similar effect in other ways (e.g., tail -f, or terminal scrollback), but I find tee is convenient.
As seen on https://unix.stackexchange.com/questions/25372/turn-off-buff... , either `unbuffer long_running_thing` or `stdbuf -oL long_running_thing` causes it to output linewise instead of 4k at a time.
[1]: http://fishshell.com/
dag from GitHub came up with[0] a quick way of using psub:
[0]: https://github.com/fish-shell/fish-shell/issues/719#issuecom...moreutils http://joeyh.name/code/moreutils/