News for hackers. Stuff that matters.

A comment on How is GNU `yes` so fast?.

Re:How is GNU `yes` so fast? by mkj on Tuesday June 13, 2017 @02:09AM (#14543119)
You could make "yes" faster with the tee() syscall. Keep duplicating data from the same fdin (doesn't actually copy) and it becomes entirely zero-copy.
[ Parent | on HN ]
Re:How is GNU `yes` so fast? by mrb on Tuesday June 13, 2017 @03:52AM (#14543425)
Actually yes should use vmsplice().

And pv on the other side of the pipe should use splice().

Now that would be a complete zero-copy I/O path, purely limited by the CPU, not by memory bandwidth. It would benchmark at hundreds of GB/s :)

[ Parent | on HN ]
Re:How is GNU `yes` so fast? by gpderetta on Tuesday June 13, 2017 @08:14AM (#14544197)
Someone posted an implementation which uses vmsplice on the reddit thread:

https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

[ Parent | on HN ]
Re:How is GNU `yes` so fast? by mrb on Tuesday June 13, 2017 @08:44AM (#14544317)
His code can further be optimized by having only 1 large element in the iovec (vs. many 2-byte "y\n" elements).
[ Parent | on HN ]
Re:How is GNU `yes` so fast? by nyanscat on Tuesday June 13, 2017 @09:46AM (#14544647)
from what i can see it's already in a large iovec buffer
[ Parent | on HN ]
Re:How is GNU `yes` so fast? by mrb on Tuesday June 13, 2017 @09:49AM (#14544659)
Correct, I misread the code. However bumping the pipe size buffer bumps the speed from 28 to 74 GB/s on my Skylake CPU (i5-6500).

Edit: I optimized further and now get 123 GB/s See https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

[ Parent | on HN ]
Re:How is GNU `yes` so fast? by gpderetta on Tuesday June 13, 2017 @12:24PM (#14545981)
next step is playing tricks with memory mapping so that the buffer is large in virtual memory but only takes a single page in physical memory, fitting in L1 (at least on the read side).
[ Parent | on HN ]
Re:How is GNU `yes` so fast? by qb45 on Tuesday June 13, 2017 @06:01AM (#14543797)
> And pv on the other side of the pipe should use splice().

Good catch, I just noticed that it does.

[ Parent | on HN ]
Re:How is GNU `yes` so fast? by masklinn on Tuesday June 13, 2017 @04:12AM (#14543494)
On the other hand the current code is perfectly portable, tee(2) is a linux syscall.
[ Parent | on HN ]
Re:How is GNU `yes` so fast? by AstralStorm on Tuesday June 13, 2017 @03:04AM (#14543292)
Still gets piped and hits same performance.
[ Parent | on HN ]
Re:How is GNU `yes` so fast? by mkj on Tuesday June 13, 2017 @03:08AM (#14543304)
It's 2x times faster for a quick test here. Only copying on the read side?
[ Parent | on HN ]
Re:How is GNU `yes` so fast? by readittwice on Tuesday June 13, 2017 @06:10AM (#14543823)
Out of interest: Could you please post the code?
[ Parent | on HN ]
Re:How is GNU `yes` so fast? by mkj on Tuesday June 13, 2017 @09:13AM (#14544451)
https://pastebin.com/jrcJbjU4

I've realised straight tee() is actually wrong - it works fine for piping to something but "zcyes > /dev/null" won't work, it needs vmsplice() like mrb said.

  core2duo e8400, 4.8 kernel, coreutils 8.25
  > yes | dd bs=1M count=10000 of=/dev/null iflag=fullblock
  10485760000 bytes (10 GB, 9.8 GiB) copied, 4.1901 s, 2.5 GB/s
  > ./zcyes | dd bs=1M count=10000 of=/dev/null iflag=fullblock
  10485760000 bytes (10 GB, 9.8 GiB) copied, 1.8542 s, 5.7 GB/s
[ Parent | on HN ]