News for hackers. Stuff that matters.
Re:How is GNU `yes` so fast? by IshKebab on Tuesday June 13, 2017 @05:02AM (#14543639)
I totally disagree. Nobody will ever want to use `yes` at 10 GB/s. They will want it to be reliable, and this sort of over-optimisation increases the risk of bugs.
Re:How is GNU `yes` so fast? by joosters on Tuesday June 13, 2017 @05:48AM (#14543757)
I've used 'yes' many times to generate huge amounts of data quickly. Back then, it never had the small string optimisation, but you could always run 'yes InsertReallyLongStringHere' to spew out data much faster than /dev/urandom or even /dev/zero

I'm glad it runs fast, and I hope that all OS utilities are optimised (and tested, of course!) instead of making their source code pretty. The fact is, most people want to use programs, not read them.

Re:How is GNU `yes` so fast? by currysausage on Tuesday June 13, 2017 @09:18AM (#14544475)
> The fact is, most people want to use programs, not read them.

I want to use safe programs, and programs with readable code are more likely to be properly audited.

Re:How is GNU `yes` so fast? by joosters on Tuesday June 13, 2017 @09:23AM (#14544503)
Audited UNIX tools - does such a thing exist?
Re:How is GNU `yes` so fast? by _cx2w on Tuesday June 13, 2017 @09:28AM (#14544540)
openbsd.org
Re:How is GNU `yes` so fast? by khedoros1 on Tuesday June 13, 2017 @01:45PM (#14546832)
Sounds like you want to stick to something with a non-GNU userspace, apparently.
Re:How is GNU `yes` so fast? by masklinn on Tuesday June 13, 2017 @07:06AM (#14543965)
> you could always run 'yes InsertReallyLongStringHere' to spew out data much faster than /dev/zero

That really doesn't make any sense, /dev/zero should be at least as fast as yes.

Re:How is GNU `yes` so fast? by joosters on Tuesday June 13, 2017 @07:19AM (#14544002)
/dev/zero should be at least as fast as yes

I agree, all I remember is that when I tried it, /dev/zero sometimes sucked performance-wise. I can't recall the exact circumstances as it was some time ago, and could have been on any of Linux/FreeBSD/SunOS/HP-UX/IRIX - perhaps it was the fastest common way at the time?

On a recent x64 Linux, /dev/zero seems plenty fast enough now:

  $ dd bs=8k count=819200 if=/dev/zero of=/dev/null
  819200+0 records in
  819200+0 records out
  6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.331137 s, 20.3 GB/s

  $ yes | dd bs=8k count=819200 of=/dev/null
  819200+0 records in
  819200+0 records out
  6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.959551 s, 7.0 GB/s
Re:How is GNU `yes` so fast? by Tepix on Tuesday June 13, 2017 @09:08AM (#14544425)
No need for "dd", let "pv" get the data from /dev/zero directly:

$ pv < /dev/zero > /dev/null [ 16GiB/s]

But the version of yes using vmsplice() is even faster than that on my machine.

Re:How is GNU `yes` so fast? by pbhjpbhj on Tuesday June 13, 2017 @09:55AM (#14544686)
What's the line to test `yes` with `pv`?

    pv < /usr/bin/yes > /dev/null
doesn't seem to work properly. FWIW I get 330MiB/s vs 8.4GiB/s for /dev/zero.

[Incidentally first I've heard of pv but I've known about dd for a decade or two].

Re:How is GNU `yes` so fast? by masklinn on Tuesday June 13, 2017 @11:37AM (#14545579)
< and > are for file redirection, yes is a binary so you want to pipe its stdout into pv:

    yes | pv > /dev/null
Re:How is GNU `yes` so fast? by pbhjpbhj on Wednesday June 14, 2017 @09:52AM (#14552678)
So how did that redirect even work; should we be doing a `mknod` to make a "yes" device to make the comparison work (can we, does it help other than in my naive imagination).
Re:How is GNU `yes` so fast? by BenjiWiebe on Wednesday June 14, 2017 @02:26PM (#14554824)
The reason that redirect worked is because it was using the contents of the yes program instead of its output.
Re:How is GNU `yes` so fast? by kevindqc on Tuesday June 13, 2017 @03:39PM (#14547911)
I know this works, but how come we can see the output of pv when it is redirected to /dev/null? Maybe I just don't understand how pipes and redirection works since I rarely use Linux :(
Re:How is GNU `yes` so fast? by masklinn on Wednesday June 14, 2017 @03:57AM (#14551286)
> I know this works, but how come we can see the output of pv when it is redirected to /dev/null?

From pv's man page:

> Its standard input will be passed through to its standard output and progress will be shown on standard error.

> Maybe I just don't understand how pipes and redirection works since I rarely use Linux :(

The Windows/DOS command line has the same concepts[0], though it's probably used less often: by default a process has 3 FDs for STDIN (0), STDOUT (1) and STDERR (2).

At a shell

* you can feed a file to STDIN via <$FILE (so `</dev/zero pv` will feed the data of /dev/zero to pv), the 0 is optional

* you can pipe an output to a command (other | command) or the output of a command to an other (command | other)

* you can redirect the STDOUT or STDERR to files via 1> and 2> (the "1" is optional so 1> and > are the same thing) (you can redirect both independently)

* you can "merge" either output stream to the other by redirecting them to &FD so 1>&2 will redirect STDOUT (1) to STDERR (2) and 2>&1 will redirect STDERR (2) to STDOUT (1), you can combine that with a regular redirection so e.g. `command >foo 2>&1 ` or with a pipe (`command 2>&1 | other`)

And you can actually create more FDs to use in your pipelines[1] though I don't remember ever seeing

[0] https://support.microsoft.com/en-us/help/110930/redirecting-...

[1] http://tldp.org/LDP/abs/html/io-redirection.html

Re:How is GNU `yes` so fast? by wutbrodo on Tuesday June 13, 2017 @03:54PM (#14548043)
You're redirecting stdout to /dev/null and pv is writing to stderr. If you use &> instead, stderr and stdout will both be redirected to /dev/null and you will see no output at all.
Re:How is GNU `yes` so fast? by moosingin3space on Tuesday June 13, 2017 @10:13AM (#14544805)
Maybe this?

    yes | pv > /dev/null
Re:How is GNU `yes` so fast? by wutbrodo on Tuesday June 13, 2017 @03:56PM (#14548064)
That's not quite apples-to-apples though:

cat /dev/zero | pv > /dev/null

uses a pipe like the 'yes' line above, but runs substantially slower on my computer than the redirect-only version.

Re:How is GNU `yes` so fast? by MichaelBurge on Tuesday June 13, 2017 @08:50AM (#14544342)
'yes' to a file is how I sometimes benchmark disk speed. It should have fewer system calls than reading from /dev/zero and then writing.

I actually checked just now, and it looks like you'd be making twice as many system calls with /dev/zero compared to generating the data locally:

    strace dd count=1 bs=512 if=/dev/zero of=test
    open("/dev/zero", O_RDONLY)             = 3
    dup2(3, 0)                              = 0
    close(3)                                = 0
    lseek(0, 0, SEEK_CUR)                   = 0
    open("test2", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
    dup2(3, 1)                              = 1
    close(3)                                = 0
    read(0,"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
    write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
    close(0)                                = 0
    close(1)    
While 'strace yes > test2' is just a constant stream of write() calls.

The difference matters if you're benchmarking e.g. some new SSD compared to a tmpfs on a machine with 100+ GB of RAM. It's always better if the tools have less overhead, because the comparison is more meaningful.

Also consider that it can be faster to write to a local network than to disk. I've never done it, but I imagine that the kernel's not going to want to deal with your /dev/zero calls if it's spending all of its time writing to a 10GB switch. I can imagine some very specialized storage servers that could spend most of their time writing from memory buffers to a network switch, or if you're troubleshooting a slowdown in the networking itself.

Re:How is GNU `yes` so fast? by wolfgang42 on Tuesday June 13, 2017 @10:18AM (#14544839)
When I started this comment, I didn't think you were measuring what you thought you were measuring with those straces. 'strace yes > test2' only watches 'yes', not '> test2' (which is handled by the shell). Here's what your command outputs:

    $ strace yes > /tmp/test2
    [various initialization steps]
    write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    [...]
To measure everything, I started up a whole new shell, expecting to see a 'write(1, ...)', a 'read(1)', and a 'write(f, ...)':

    $ strace -f sh -c 'yes > /tmp/test2'
    [various initialization steps]
    [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    [...]
How does this possibly work? File descriptor 1 is supposed to be the terminal, not a file! Of course, the magic of file redirection:

    open("/tmp/test2", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 # Open the file, get FD 3
    fcntl(1, F_DUPFD, 10)                   = 10 # Copy FD 1 (the terminal, STDOUT) to FD 10 temporarily
    close(1)                                = 0  # Close original FD 1
    fcntl(10, F_SETFD, FD_CLOEXEC)          = 0  # Close FD 10 (the terminal, copy of STDOUT) when exec() is called
    dup2(3, 1)                              = 1  # Copy FD 3 (the file) to FD 1 (STDOUT)
    close(3)                                = 0  # Close FD 3 (the file's original descrptor)
Re:How is GNU `yes` so fast? by kccqzy on Tuesday June 13, 2017 @02:41PM (#14547410)
I had the impression that you thought "yes" pipes the output to shell, and then the shell writes to disk. That's incorrect. " > file" means redirecting to a file, and therefore all write system calls actually write to disk.
Re:How is GNU `yes` so fast? by wolfgang42 on Tuesday June 13, 2017 @03:22PM (#14547781)
That's exactly what I thought. Now, had you asked me how redirection worked I would have said "the redirection operators cause the shell to attach a file to the descriptor," but I'd never actually thought through the implications of that in terms of what syscalls get made, and the output of strace presented a rather visceral demonstration of the implications of this clever bit of design.
Re:How is GNU `yes` so fast? by yk on Tuesday June 13, 2017 @05:20AM (#14543685)
The ability to get 10 GB/s dummy date into a pipe from the command line could come in handy at some point, for stress testing or something. I am not sure if it is over-optimized. (And even with the optimizations the risk of bugs should be very small.)
Re:How is GNU `yes` so fast? by masklinn on Tuesday June 13, 2017 @07:05AM (#14543961)
Of course you could just pipe from /dev/zero which will easily do 10GB/s on every machine.
Re:How is GNU `yes` so fast? by yk on Tuesday June 13, 2017 @08:31AM (#14544261)
Just tried it,

    cat /dev/zero|pv >/dev/null 
gives me roughly 2/3 the speed of yes. (5 GB/s vs 7 GB/s) Plus yes gives you an arbitrary string, instead of only zeroes.
Re:How is GNU `yes` so fast? by Tepix on Tuesday June 13, 2017 @09:09AM (#14544436)
Tobik already wrote it

but here's todays Useless Use of Cat Award http://porkmail.org/era/unix/award.html

Re:How is GNU `yes` so fast? by e12e on Wednesday June 14, 2017 @06:27AM (#14551745)
Well, sometimes using cat is faster, as I discovered to my surprise recently:

https://news.ycombinator.com/item?id=14414610

But in that case one could probably say that Gnu awk is just not very good at input handling (as Mawk doesn't appear to benefit from an extra "cat").

Re:How is GNU `yes` so fast? by tobik on Tuesday June 13, 2017 @08:43AM (#14544313)
That's because you're using cat and a pipe. Try this instead:

  pv > /dev/null < /dev/zero
Re:How is GNU `yes` so fast? by mikeash on Tuesday June 13, 2017 @10:04AM (#14544742)
For anyone who is like me and finds it uncomfortable that things are now out of order, note that you can still put the input redirection in front:

  </dev/zero pv >/dev/null
Re:How is GNU `yes` so fast? by vram22 on Tuesday June 13, 2017 @06:37PM (#14549260)
Or even this:

pv </dev/zero >/dev/null

which is a common way of doing it (for any command with any inputs and outputs, not just the above ones), i.e.:

command < input_source > output_dest

All three pv command invocation variants, the one here and the two above, work. And it becomes more clear why they are the same, when you know that the redirections are done by the shell + kernel, and the command (such as pv) does not even know about it. So in all three cases, it does not see the redirections, because they are already done by the time the command starts running (with its stdin and stdout redirected to / from those respective sources). And that is precisely why the command works the same whether it is reading from the keyboard or a file or pipe, and whether it is writing to the screen or a file or pipe.

Re:How is GNU `yes` so fast? by duozerk on Tuesday June 13, 2017 @09:35AM (#14544580)
I believe /dev/zero writes data one byte at a time; that's likely the reason why.

[edit] That's actually inaccurate (and badly expressed), see comments below.

Re:How is GNU `yes` so fast? by adwn on Tuesday June 13, 2017 @10:33AM (#14544966)
/dev/zero doesn't "write" anything in the sense that yes writes, since it's a character device and not a program. The Linux kernel's implementation of /dev/zero does not write one byte at a time.
Re:How is GNU `yes` so fast? by duozerk on Tuesday June 13, 2017 @10:38AM (#14545005)
You're right of course; and actually I believe the kernel will simply provide as many bytes as the read() requested; so the speed should mostly depend on how you access /dev/zero. IE, the user above was using cat and I think with dd and a proper block size it'd be much faster.
Re:How is GNU `yes` so fast? by sqeaky on Tuesday June 13, 2017 @12:08PM (#14545831)
I was under the impression that cat automatically used a sane size for reading. Now that I think of it I cannot think of a source, other that to point out my own anecdotal experience.

When I was writing raspbian images to SD cards to use on a raspabery Cat and DD took within a few seconds of each other on an operation longer then a minute. Since then I have been using cat where I could, but I didn't think to right down the numbers though.

Re:How is GNU `yes` so fast? by e12e on Wednesday June 14, 2017 @06:33AM (#14551777)
For something somewhat related, see parts of this thread:

https://news.ycombinator.com/item?id=14414610

Note that cat+gnu awk was faster than just gnu awk - but mawk was faster still (reading a not entirely small file).

And in a similar vein of gp comparing Gnu and Openbsd, note that openbsd cat is a little more convoluted than the simplest possible implementation (at least to my eyes):

https://github.com/openbsd/src/blob/master/bin/cat/cat.c

https://github.com/coreutils/coreutils/blob/master/src/cat.c

(That is, Gnu "cat" and OpenBSD "cat" are less different than Gnu "yes" and OpenBSD "yes").

Re:How is GNU `yes` so fast? by problems on Tuesday June 13, 2017 @09:28AM (#14544539)
yes will give repeated data though, not just zeros, seems like it's more useful here - as others have pointed out, also uses less syscalls than /dev/zero.
Re:How is GNU `yes` so fast? by belorn on Tuesday June 13, 2017 @07:33AM (#14544046)
If you want to maximize readability and simplicity then writing `yes` in C is a bad choice. It is much easier, cleaner and shorter to write it in python or just use the shell which would normally be using the `yes` in the first place. Since `yes` is used in shells and builtins can be considered very reliable, here is a implementation as a shell function:

    yes(){ while :; do echo "${1:-y}"; done; }
Python:

    import sys 
    while True:
      if len(sys.argv) > 1:
        print(sys.argv[1])
      else:
        print("y")
And if you don't need the features of `yes` and only need a loop that prints 'y', then there really is hard to beat the simplicity of:

  while:; do echo "y"; done
Re:How is GNU `yes` so fast? by raptorfactor on Tuesday June 13, 2017 @07:42AM (#14544076)
Is it really "easier, cleaner and shorter to write it in python"? Did you look at the OpenBSD implementation?

https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c

It's essentially line for line identical to your python code...

Re:How is GNU `yes` so fast? by belorn on Tuesday June 13, 2017 @09:38AM (#14544599)
The C example has three includes, two conditionals, two loops, and one function definition. The python example has a single include, conditional, and loop.

For readability purpose it is easier to go through each lines of the python program than the OpenBSD C code. Its not massively different, but its distinguishable enough that I would choose the python version if I wanted to maximize readability, minimize syntax requirement and did not want to use shell script.

The Shell function is in my view the superior choice if the audience is a programmer than know the shell script syntax. It is just a single loop and is written in the environment that the program is intended to be used in. The only drawback there is the speed.

Re:How is GNU `yes` so fast? by mikeash on Tuesday June 13, 2017 @10:09AM (#14544787)
Most of what makes the C program bigger comes from the fact that the C program does more. Your python example doesn't call pledge(). Remove that from the C program and it drops to one include, one conditional, and two loops. Further, counting the two loops against C doesn't make any sense: it's entirely up to the programmer whether to have a conditional containing two loops, or a loop containing a conditional. Both languages could naturally do it either way.
Re:How is GNU `yes` so fast? by teamhappy on Tuesday June 13, 2017 @11:08AM (#14545314)
> The python example has a single include, conditional, and loop.

... and python. Don't forget to count python.

Re:How is GNU `yes` so fast? by BenjiWiebe on Wednesday June 14, 2017 @02:49PM (#14555009)
Exactly. That's the reason I write stuff in C instead of my favorite interpreted language, Ruby. When you write something in C, that's it. No large interpreter plus runtime needed.
[dead]
Re:How is GNU `yes` so fast? by sctb on Tuesday June 13, 2017 @01:27PM (#14546615)
You've been on a tear of uncivil and unsubstantive commenting, and it has to stop. Often a good strategy for this is to slow down. High-quality reflective posts are what we're after instead of dismissive, snarky reflexive ones, and the former come more slowly.
Re:How is GNU `yes` so fast? by _RPM on Tuesday June 13, 2017 @09:36AM (#14544591)
This is innificient. Why do the argv check in every iteration of the while loop? It's not going to change between iterations.
Re:How is GNU `yes` so fast? by th4dv on Tuesday June 13, 2017 @08:10AM (#14544180)
> Nobody will ever want to use `yes` at 10 GB/s

Just because you say so?

Re:How is GNU `yes` so fast? by bastawhiz on Tuesday June 13, 2017 @12:26PM (#14545996)
If the speed of yes is bounded by memory speed, doing anything useful will almost certainly consume that data at a far lower rate. Putting it on a disk, pushing it over a network, etc. will almost always be slower than yes is able to generate data.
Re:How is GNU `yes` so fast? by cestith on Tuesday June 13, 2017 @01:20PM (#14546543)
The typical use case is piping it into another running program. Maybe someone wants to do that really quickly rather than putting it on disk or pushing it over a network.
Re:How is GNU `yes` so fast? by tomohawk on Tuesday June 13, 2017 @05:45PM (#14548938)
It's not about running 'yes' at 10GB/s, it's about less overhead to do a simple job. If this version of yes is 100x faster, that implies it using 1% of a cpu to do the same work that would otherwise occupy 100% of a cpu. This leaves more of the machine to do what is likely to be the intended task.
Re:How is GNU `yes` so fast? by peterwwillis on Tuesday June 13, 2017 @02:10PM (#14547121)
Unix userland tools have evolved over decades to be as efficient as possible because they have historically underpinned everything the operating system does. The faster each tool works, the faster the other tools that depend on them work. If increased efficiency results in a bug, that bug can then be fixed, making it a net gain for system stability.
Re:How is GNU `yes` so fast? by avar on Tuesday June 13, 2017 @05:07AM (#14543646)
I'm having a hard time imagining a case where even a grossly complex 100,000 line implementation of yes(1) couldn't be trivially proven to be correct.
Re:How is GNU `yes` so fast? by mikeash on Tuesday June 13, 2017 @10:12AM (#14544803)
Easy! Write a program that does a brute force check of Goldbach's conjecture on all integers. For each integer that passes the check, print a line. If you can prove this correct (or incorrect) you'll probably win a Fields medal.
Re:How is GNU `yes` so fast? by acdha on Tuesday June 13, 2017 @08:37AM (#14544286)
I agree with the general point that you could prove such a simple program correct relatively easily but that does still have a cost, which is always a concern in an open-source project. You still need someone to step up and do that work and continue to verify it periodically in the future – if that code is doing complicated things with buffering, that opens up possible odd bugs due to stdlib, gcc, maybe even kernel behaviour changes which might not affect simpler programs.

Not a huge bit of work to be sure but for a non-commercial project you might have trouble finding a volunteer who cares about that tedium.

Re:How is GNU `yes` so fast? by fredmorcos on Tuesday June 13, 2017 @07:51AM (#14544106)
Absolutely, I once trivially proved a 1000,000 line implementation of return 0 to be correct. I don't know why all comments are bothered by how much overkill this yes implementation is. Maybe they don't hold 100,000 PhDs like we do, am I right?
Re:How is GNU `yes` so fast? by nobodyorother on Tuesday June 13, 2017 @02:04PM (#14547039)
Hey, 640K is enough for anyone!