When your data doesn’t fit in memory: the basic techniques

Re:When your data doesn’t fit in memory: the basic techniques by scarejunba on Monday November 11, 2019 @05:13PM (#21509456)

I'm actually pretty much against UUoC. The cat example is much more resilient to modification. For instance, there are usually natural modifications considering filtering etc.

Anyway, since he said number we probably want `nl` not `wc`.

Separating pipe-setup from processing is sound engineering to me and makes things much easier. For instance as I iterate, I will have cat piped to head piped to filter but eventually I'll take the head out and run the whole thing. That's a trivial ^W versus moving around arguments and worrying about arg order, etc.

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by navaati on Monday November 11, 2019 @05:26PM (#21509570)

That's right regarding directly giving the filename for the program (here wc) to open itself.

However using the input redirect is fine, especially if you format it that way (which is exactly the same thing):

  <file wc -l

Then, adding another step is as natural as with cat:

  <file grep meh |wc -l

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by thestoicattack on Monday November 11, 2019 @05:34PM (#21509641)

Useless use of `wc` :)

    <file grep -c meh

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by scarejunba on Monday November 11, 2019 @06:19PM (#21509990)

25 years using shells and I never thought to place input redirection at the front. Well, thank you!

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by herpderperator on Monday November 11, 2019 @05:36PM (#21509667)

I am confused. What's wrong with `wc -l < file | ... | ...`?

I know `< file wc -l | ... | ...` is identical because the redirection is done before the command is executed, but what does putting it in front help with?

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by squiggleblaz on Tuesday November 12, 2019 @05:23AM (#21513006)

Because I feel the other answers, though correct, are entirely missing out on clarity:

The first iteration looks like this:

<file head -n10 | grep foo | tr baz qux

The second iteration looks like this:

<file gzcat | head -n10 | grep foo | tr baz qux

It makes the change much easier, since the first thing written is the first thing that happens. In `gzcat < file`, the logical first step - reading and streaming file - is now the physical second step. Like a German sentence, whenever you want to prepend the file, you have to maintain an item in second position.

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by emj on Monday November 11, 2019 @05:52PM (#21509789)

You can easily change that first command with muscle memory. That's why I use cat $file| but I guess <$file would work too.

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by emmelaich on Monday November 11, 2019 @07:12PM (#21510283)

As per James Taylor's comment above, using cat means you can easily replace it with head e.g. for testing.

And not change anything else in the pipeline.

[ Parent | on HN ]