News for hackers. Stuff that matters.
Re:Stop Piping Cats by jrockway on Wednesday February 10, 2010 @05:07PM (#1116183)
This is my least-favorite Internet meme. People "pipe cats" because they want the entire pipeline to read left-to-right, like:

   cat file | xargs foo | grep bar | sort | wc -l
It just looks nicer than:

   < file xargs foo | grep bar ...
Re:Stop Piping Cats by gwern on Wednesday February 10, 2010 @05:11PM (#1116195)
I would also add that not all tools follow the '-' convention, and the ones that do can break in corner-cases. Why invest the effort of remembering how every tool works and which ones don't, for a obfuscating micro-optimization? /proud useless user of cat
Re:Stop Piping Cats by jrockway on Wednesday February 10, 2010 @05:21PM (#1116224)
Exactly. I tried to figure out how to make xargs read from a file instead of stdin, and was unsuccessful. My "cat ... |" works every time (at the expense of 5 miliseconds of CPU time. oh noes.)
Re:Stop Piping Cats by camccann on Wednesday February 10, 2010 @07:43PM (#1116546)
There's a nice consistency in doing it that way--it's very easy to think of cat as turning an "inert" file into a stream of data, each program as a function that applies a transformation to a stream with a particular kind of content, and the pipe as function composition (given compatible stream content). A very comfortable programming-like perspective, at least to me. (Pop quiz: What programming idiom closely resembles this "repackage content and apply sequential transformations" approach?)
Re:Stop Piping Cats by silentbicycle on Thursday February 11, 2010 @08:08AM (#1117508)
Spoiler alert: See also Oleg Kiselyov's "Monadic i/o and UNIX shell programming" (http://okmij.org/ftp/Computation/monadic-shell.html).
Re:Stop Piping Cats by theblackbox on Wednesday February 10, 2010 @07:57PM (#1116569)
Seriously?

You failed at that?

I'm intrigued because it always occured to me that you were a particularly fastidious person (with all due respect).

I often come accross issues that would seem for all the world to be a pointless artifact of an accepted convention. If this accepted convention is an acquired trait and one learns exclusivly from a limited meduim (irc frustrates me), I find it very curious that they persist as they sometimes act as a difficult barrier for a thorough understanding. This xargs thing, though I know almost nothing about it, seems to parallel similar problems I've encountered and I for one would be interested to hear your take on this (and HN in general).

Or just some food for thought for your next blog post maybe?

Re:Stop Piping Cats by jrockway on Wednesday February 10, 2010 @08:21PM (#1116603)
Well, my guess is that xargs simply does not accept a filename as the argument. This comes from reading the --help and grepping the manpage for "file".

The point was, when I used "cat file |", I knew it was going to work, and it did. When I tried to eliminate the cat by using the program's built-in ability to read a file, I had to read several manpages before determining that it was not possible. All because some tutorial says "cat is useless", when it clearly saved me much more time than the extra CPU time it used.

And if you are actually asking; xargs is just a utility to read command-line-arguments from stdin. `echo -e "foo\nbar" | xargs rm' === `rm foo; rm bar' (or depending on the xargs implementation; `rm foo bar'). It kind of reminds me of a "functor map" operation, where stdin is a functor (of command-line arguments), and command-line programs are functions. (I will now mention that xargs also does "join" on the "results" of the "function", which is very ... monad-like. But "Monads are teh awesome and everything is one" is my second-least-favorite Internet meme, so I will spare you. :)

Re:Stop Piping Cats by olefoo on Thursday February 11, 2010 @01:07AM (#1117032)
The whole 'useless use of cat' meme is basically designed to let Randal Schwartz make fun of people. At one point it may have made a difference, and certainly if you have a shell script that is getting called 10,000 times a day it might make sense to optimise it. But for doing stuff from the commandline; weird shell acrobatics is a premature optimisation.
Re:Stop Piping Cats by jrockway on Thursday February 11, 2010 @01:30AM (#1117068)
The whole 'useless use of cat' meme is basically designed to let Randal Schwartz make fun of people.

So true.

My requirement for a shell command is that it be reasonably easy to assemble, return something resembling the correct answer, and that it run in a reasonable amount of time.

If your requirements are more strict than those, it's time to write a real program.

Re:Stop Piping Cats by Estragon on Wednesday February 10, 2010 @11:40PM (#1116916)

  I tried to figure out how to make xargs read from a file 
  instead of stdin, and was unsuccessful.
"xargs -a <filename>" First option described in the man page on Red Hat.
Re:Stop Piping Cats by jrockway on Thursday February 11, 2010 @01:56AM (#1117115)
My home machine (Debian GNU/Linux) says "--arg-file", and it looks like a relatively new feature. My RHEL box at work definitely didn't have anything with the word "file" in it.
Re:Stop Piping Cats by gwern on Thursday February 11, 2010 @11:08AM (#1117894)
The whole thread descending from here is an excellent example of what I mean by corner-cases and memorization. :)
Re:Stop Piping Cats by lg on Wednesday February 10, 2010 @11:52PM (#1116931)
could be a GNU vs. BSD thing.
Re:Stop Piping Cats by silentbicycle on Thursday February 11, 2010 @08:07AM (#1117506)
BSD xargs doesn't seem to have the -a option.
Re:Stop Piping Cats by spudlyo on Wednesday February 10, 2010 @05:31PM (#1116248)
Sometimes you can use /dev/fd/0, but I agree, it's a stupid micro optimization, and I don't understand why people get so righteous about it. My co-worker would correct you every time he saw it, but then again he's a pedantic geek.
Re:Stop Piping Cats by pyre on Wednesday February 10, 2010 @06:06PM (#1116320)
It's still good to keep it in mind, in case there are instances where it's being executed many times causing a bottleneck (though one-liners aren't really meant to be used in such situations).
Re:Stop Piping Cats by wendroid on Thursday February 11, 2010 @05:54AM (#1117399)
zsh and rc provide primitives for this, this is rc

    diff <{ps} <{sleep 3; ps}
Re:Stop Piping Cats by vidarh on Thursday February 11, 2010 @10:49AM (#1117838)
Equivalent in bash:

    diff <(ps) <(sleep 3; ps)
Re:Stop Piping Cats by blasdel on Wednesday February 10, 2010 @05:24PM (#1116230)
Speak for yourself, and I don't know anyone that puts the input redirection first. I do this:

  (xargs foo | grep bar | sort | wc -l) < file
It makes the pipeline one command, both lexically (verb comes first) and concretely (the pipeline is kicked off in a subshell)
Re:Stop Piping Cats by barrkel on Wednesday February 10, 2010 @05:51PM (#1116288)
At the expense of the data flowing from right to left to right, from the outside in. At least with cat, data flows unambiguously from left to right.
Re:Stop Piping Cats by blasdel on Wednesday February 10, 2010 @08:54PM (#1116649)
But using cat totally fucks up the calling conventions: it puts the operand as the second of many arguments. Which one of these doesn't belong?:

  verb file
  cat file | verb1 | verb2 | verb3
  (verb1 | verb2 | verb3) < file
  verb123 file
EDIT: Consider the calling conventions, which one of these handles it's arguments differently? Assume that verb123 is an equivalent to the pipeline -- the subshell+stdin construction lends itself to shell aliases:

  alias verb123="(verb1 | verb2 | verb3) < "
Re:Stop Piping Cats by barrkel on Wednesday February 10, 2010 @09:04PM (#1116659)
I have no idea which one doesn't belong. Is 'verb123' the moral equivalent of '(verb1 | verb2 | verb3)'? In that case, it's the first one. But that's visible from the first word on the line, so I'm still not getting the picture.

Of course, with the usual argument vs stdin conventions, either of the middle two could also be rewritten:

    verb1 file | verb2 | verb3
This is probably how I'd write such a chain.
Re:Stop Piping Cats by jrockway on Wednesday February 10, 2010 @10:15PM (#1116783)
Spoken like a Haskell programmer, who might write:

   (wc Lines) . sort . (grep "bar") . (xargs "foo") =<< file
But note in this case that all the data "flows in the same direction".
Re:Stop Piping Cats by blasdel on Thursday February 11, 2010 @02:58AM (#1117179)
has been a Haskell programmer since 2005, before the dawn of dons

  wc Lines $ sort $ grep "bar" $ xargs "foo" =<< file
Re:Stop Piping Cats by jrockway on Friday February 12, 2010 @02:45PM (#1121574)
That does not parse the way you think it does; the fixity of =<< is greater than $. You would need the shell-style parens if you insist on application instead of composition :)

    Prelude> :info $
    ($) :: (a -> b) -> a -> b 	-- Defined in GHC.Base
    infixr 0 $
    Prelude> :info =<<
    (=<<) :: (Monad m) => (a -> m b) -> m a -> m b
  	-- Defined in Control.Monad
    infixr 1 =<<
    Prelude> :info .
    (.) :: (b -> c) -> (a -> b) -> a -> c 	-- Defined in GHC.Base
    infixr 9 .
Incidentally, the parens in my example are actually unnecessary. Function application is about 10, and (.) is 9.
Re:Stop Piping Cats by gnosis on Thursday February 11, 2010 @12:02AM (#1116945)
I almost never use <

It's too easy to make a mistake and type >

..and that can ruin your whole day.

Re:Stop Piping Cats by blasdel on Thursday February 11, 2010 @02:52AM (#1117170)
I have fucked myself more than once by typing > instead of >>

I find that to be a far more pernicious design error -- they should have made the longer token the destructive one, or used another character in it.

Re:Stop Piping Cats by thornist on Thursday February 11, 2010 @11:45AM (#1117996)
zsh with 'setopt noclobber' won't overwrite an existing file with '>', instead requiring '>|'. The history recall conveniently fills in the '|' for you if you up-arrow after a failed call. It works pretty nicely - I imagine other shells have similar.
Re:Stop Piping Cats by pyre on Wednesday February 10, 2010 @06:08PM (#1116328)
But does the time it takes to launch the subshell equal the amount of time it takes to launch cat? If so your improvement is a NOOP.
Re:Stop Piping Cats by blasdel on Wednesday February 10, 2010 @09:08PM (#1116669)
fork + 3x(fork+exec) is going to be cheaper than 4x(fork+exec), especially if cat isn't resident.

The point is a logical improvement anyway (not burying the input argument near the beginning). I'm kind of surprised that the bash folks haven't turned cat into a builtin like they did with time and some of the other coreutils.

It's too bad it's about 30 years too late to stem the tide of shit like cat -v: http://harmful.cat-v.org/cat-v/

Re:Stop Piping Cats by pyre on Thursday February 11, 2010 @01:15AM (#1117046)
By the reasoning in your link, your suggestion of bash making 'cat' one of the built-in commands is the "cancer that's bloating UNIX."
Re:Stop Piping Cats by blasdel on Thursday February 11, 2010 @03:24AM (#1117208)
The shell has always been the kitchen sink full of glue holding the whole thing together. There have always been builtins: language control structures, job control, etc. -- there are some things you can't trust others not to fuck up, and where the coupling would just get ridiculous.

Here's cat as a pure sh builtin:

  shcat {
    for arg in "$@"; do
      exec 3<>"$arg"
      while read line <&3; do
        echo "$line"
      done
      exec 3>&-
    done
  }
The shell by it's very nature can't just do exactly one task well, it's a programmable environment for living in. The cancer that's bloating UNIX was the way that the BSD and especially the GNU crews took simple tools and cross-pollinated them randomly with stupid shit. Try running "/bin/true --help" on a GNU system sometime -- there's a damn good reason why "your shell may have its own version of true".
Re:Stop Piping Cats by pyre on Thursday February 11, 2010 @11:37PM (#1119777)

  % /bin/true --help
  Usage: /bin/true [ignored command line arguments]
    or:  /bin/true OPTION
  Exit with a status code indicating success.

        --help     display this help and exit
        --version  output version information and exit

  NOTE: your shell may have its own version of true, which usually supersedes
  the version described here.  Please refer to your shell's documentation
  for details about the options it supports.

  Report bugs to <[email protected]>.
I wouldn't necessarily call that 'bloated' unless you feel that any program that uses one bit more than absolutely necessary should be scrapped as 'bloated beyond belief.'

> there's a damn good reason why "your shell may have its own version of true".

Because why exactly?

Re:Stop Piping Cats by l0stman on Thursday February 11, 2010 @10:43AM (#1117823)
If all you want is to group commands, you could use {...} instead of (...) so you don't have to spawn a subshell.
Re:Stop Piping Cats by rlpb on Wednesday February 10, 2010 @10:11PM (#1116771)
For ad-hoc stuff, I tend to do different things depending on where my cursor is and how I built up to a complex pipeline.
Re:Stop Piping Cats by wendroid on Thursday February 11, 2010 @05:56AM (#1117400)
You get mostly both with

    xargs foo < file | grep bar | sort | wc -l
Re:Stop Piping Cats by hernan7 on Wednesday February 10, 2010 @07:04PM (#1116476)
Exactly. On a modern computer, the gain from not piping cats is almost always negligible. I don't think most shell scripts today ever get to the point where you have to worry about inner loops and the such.

These days, if you hit a performance wall from spawning too many cats, I would think you switch to some scripting language where you have everything in one process. Premature optimization, people...

Re:Stop Piping Cats by dy9 on Wednesday February 10, 2010 @10:00PM (#1116741)
What's the rationale for piping sort into wc?
Re:Stop Piping Cats by iigs on Wednesday February 10, 2010 @10:05PM (#1116757)
That looks like it was a sample line that wasn't particularly useful, but if you do:

cat file.txt | sort | uniq | wc -l ## notice uniq

it gives you the count of unique lines. If you omit the sort in that instance it will fold duplicates together a/b/a is three lines, not two.

Re:Stop Piping Cats by gnosis on Thursday February 11, 2010 @12:04AM (#1116950)
Instead of "sort | uniq" you could just use "sort -u"
Re:Stop Piping Cats by wendroid on Thursday February 11, 2010 @05:57AM (#1117402)
There's also the argument about debugging, one can insert a tee in any stage of

   cat file | xargs foo | grep bar | sort | wc -l