Exactly. Things like the "useless use of wc -l" fall into a similar category for me. When I want to count the lines of stdout, I use "wc -l". It may not be the simplest, most efficient or fewest keystrokes, but it is extremely easy to remember and fits well into the model of a pipeline of simple single-purpose tools. I tend to use cat at the beginning of pipelines just because it keeps the flow left-to-right, and makes it much clearer what the source of the data is.

There's value in knowing some of the advanced things that the textutils and bash can do together, and sometimes there's value in using those things. I don't dispute that, but I do dispute the need to shame people who use the tools in ways that you personally don't like.

Shame or not, I'm going to keep using cat at the beginning of my pipelines unless I have a really good reason not to. If that makes some people think I'm not in the Unix wizard in-group, I don't really care.

Pipes also match my thought flow. "wc -l" is a good example.

"Ok, I want to read this file..."

    cat file

"Then I want to find foo"

    cat file | grep foo

"Oh, and I want to know how many lines of foo there are"

    cat file | grep foo | wc -l

In this trivial example, I could have inserted "-c" with only a few keystrokes, but on a 200 character string of commands it's faster and easier to tack "wc -l" on at the end. Plus I'm not strong enough with grep to do every manipulation I might want, so sometimes that grep will be followed by some perl, or a sort, etc.

If you're doing a one-off pipeline on the command line, you can even start with less!

"I want to look through this file..."

  less file

"Oh, that's long, I want to look at all the foos..."

  less file | grep foo

"How many foos are there?"

  less file | grep foo | wc -l

Try it, it works!

This is not some magic behavior of pipelines. Rather, less (like many other terminal programs) is explicitly programmed to behave differently if the output is a terminal. Specifically, the less source code has this comment in it:

"/*

* Output is not a tty.

* Just copy the input file(s) to output.

*/"

grep does this to, which is why the color-coding control characters don't mess up other commands.

FWIW, that's the exact same way I approach that sort of manipulation. Start with the most basic operation, then construct a pipeline going left to right, adding filters / operations as needed to get to the result.

This form is also nicely modular, setting up a data source at the left of the pipe in a way that's easy to modify to swap in a different kind of data source:

   cat file | grep xyz
   gzcat file | grep xyz
   xzcat file | grep xyz
   pv file | grep xyz
   curl url | grep xyz

There are many places I get data, and it sometimes changes (e.g. I decide to compress a file and read it back with xzcat instead of cat, leaving the rest of the script unchanged). Unless you give me an alternate syntax that can handle all of them, I'm not interested in the fact that a particular utility has chosen to provide its own idiosyncratic syntax to special-case only the 'cat' data source, with an -f parameter or whatever.

> setting up a data source at the left of the pipe in a way that's easy to modify to swap in a different kind of data source:

Just replace

    cat file |

with

    <file

It still keeps the data source on the left, and easily swapped out:

    <file grep xyz

I'm going to agree with you here. 99% of the time this UUOC stuff is just premature optimization. If it gets the job done it does not matter if it wastes a few milliseconds.

The problem is when I see UUOC use in fairly large scripts. I've seen load averages shoot through the roof due to numerous separate commands that are being run causing needless context switching as each set of commands is being run per line of a large datafile.

(1) Like dpcx said, dealing with that isn't premature optimization.

(2) Be careful about load averages, lots of people don't understand them. It just means the number of processes ready to run over a period of time. It is entirely possible to have a very high load average with a very low cpu utilization. A bunch of little processes that hop on and off the run queue really quickly because they only have a little bit of work to do at a time can kick the load average up way higher than a single self-contained process doing the same amount of work with the same amount of total cpu utilization. The single process wins on context switches, but modern cpus cache those in hardware so its barely a win any more.

Fixing it at that point is no longer a "premature optimization"...

Wait, what? You mean like this?:

    cat somefile |while read line
    do
      echo $line|cat -|pipe...
    done

Because otherwise I can't see how a pipeline would be started for each line of a datafile ?

Sure it's possible without running a subprocess, just open the file on some other filedescriptor.

    exec 5<words.txt
    while read line <&5 ; do
        echo "line read: $line"
    done
    exec <&5-

It's more common to see this issues with something like this:

   for f in *;
   do
      cat $f | ...
   done

Now make that a directory with thousands of files or a script digs down through through the tree.

But it's not really the "cat" that's the problem here, is it?

You could argue that people who complain about wasting a cat are ignoring the Unix philosophy of "do one thing and do it well." Moreso with wc -l vs grep -c. If I know that wc -l is how to count lines, I don't want to remember the options for counting lines built into every other command.

As an alternative, you can always rewrite:

   cat file | grep xyz

To:

    <file grep xyz

Just a cute tangent: I once wrote a shell with no built-in redirection. You executed a program named "<" for that, like your second example but with a space:

    < file grep xyz

(The trouble with this design: it doesn't generalize well to pipes.)

No? Maybe you meant "grep xyz <file" ? In what shell does your line accomplish the same thing?

[edit: Ah, I had an extra pipe in there. "<file command args" does indeed work --- "<file | command args" does not]

>In what shell does your line accomplish the same thing?

Any descended from or cloning sh, or csh, or rc… i.e. pretty much all of them.

   <file grep xyz

works in bash for one.

I agree, and like the fact that you can then read your commandline as a series of steps, going from input to output.

However, in bash (and other shells?) you can write "<file grep xyz" rather than "cat file | grep xyz".

>Mentally, I think of most Unix utilities as operating on STDIN and producing STDOUT

Yeah, that is actually very much the point. You are passing the contents of the file in to grep's STDIN. So why not just do exactly that rather than cat it first for no reason? You don't need to remember how to tell commands to read from a file, you just need to learn the basics of how your shell works.

    <file grep xyz