I get around 1.56MiB/s with that code. PHP gets 4.04MiB/s. Python gets 4.35MiB/s.

> What's also interesting is that node crashes after about a minute

I believe this is because `while(1)` runs so fast that there is no "idle" time for V8 to actually run GC. V8 is a strange beast, and this is just a guess of mine.

The following code shouldn't crash, give it a try:

    node -e 'function write() {process.stdout.write("1"); process.nextTick(write)} write()' | pv > /dev/null

More examples with Babashka and Clojure:

    bb -e "(while true (print \"1\"))" | pv > /dev/null

    clj -e "(while true (print \"1\"))" | pv > /dev/null

    clj -e "(require '[clojure.java.io :refer [copy]]) (while true (copy \"1\" *out*))" | pv > /dev/null

    clj -e "(while true (.println System/out \"1\"))" | pv > /dev/null

Versions: PHP 8.1.6, Python 3.10.4, NodeJS v18.3.0, Babashka v0.8.1, Clojure 1.11.1.1105

> I believe this is because `while(1)` runs so fast that there is no "idle" time for V8 to actually run GC. V8 is a strange beast, and this is just a guess of mine.

Not exactly: the GC is still running; it’s live memory that’s growing unbounded.

What’s going on here is that WritableStream is non-blocking; it has advisory backpressure, but if you ignore that it will do its best to accept writes anyway and keep them in a buffer until it can actually write them out. Since you’re not giving it any breathing room, that buffer just keeps growing until there’s no more memory left. `process.nextTick()` is presumably slowing things down enough on your system to give it a chance to drain the buffer. (I see there’s some discussion below about this changing by version; I’d guess that’s an artifact of other optimizations and such.)

To do this properly, you need to listen to the return value from `.write()` and, if it returns false, back off until the stream drains and there’s room in the buffer again.

Here’s the (not particularly optimized) function I use to do that:

  async function writestream(chunks, stream) {
      for await (const chunk of chunks) {
          if (!stream.write(chunk)) {
              // When write returns null, stream is starting to buffer and we need to wait for it to drain
              // (otherwise we'll run out of memory!)
              await new Promise(resolve => stream.once('drain', () => resolve()))
          }
      }
  }

ETA: I should probably note that transform streams, `readable.pipe()`, `stream.pipeline()`, and the like all handle this stuff automatically. Here’s a one-liner, though it’s not especially fast:

  node -e 'const {Readable} = require("stream"); Readable.from(function*(){while(1) yield "1"}()).pipe(process.stdout)' | pv > /dev/null

But using a longer string crashed after 23s here:

    node -e 'function write() {process.stdout.write("1111111111222222222233333333334444444444555555555566666666667777777777888888888899999999990000000000"); process.nextTick(write)} write()' | pv > /dev/null

Also, what NodeJS version are you on?

node is v10.24.0. (Default from the Debian 10 repo)

After some quick testing in a couple of versions, it seems like it got fixed in v11 at least (didn't test any minor/patch versions).

By the way, all versions up to NodeJS 12 (LTS) are "end of life", and should probably not be used if you're downloading 3rd party dependencies, as there are bunch of security fixes since then, that are not being backported.

Java has (had) weird idiosyncrasies like this as well, well it doesn't crash, but depending on the construct you can get performance degradations depending on how the language inserts safepoints (where the VM is at a knowable state and a thread can be safely paused for GC or whatever).

I don't know if this holds today, but I know there was a time where you basically wanted to avoid looping over long-type variables, as they had different semantics. The details are a bit fuzzy to me right now.

  yes | pv > /dev/null

Not sure about how yes(1) compares to the techniques presented in the linked post. Perhaps there's still room for improvement.

[1] Previous HN discussion: https://news.ycombinator.com/item?id=14542938

  pv < /dev/zero > /dev/null

yes lets you specify which character to output. 'yes n' for example to output n.

    yes 123abc

    123abc123abc123abc123abc123abc

  123abc
  123abc
  123abc
  ...

Honest question: what are the practical use cases of this?

Repeatedly typing the 'y' character into a Linux pipe is surely not that common, especially at that bit rate. Also seems like the bottleneck would always be the consuming program...

It's probably still common in installation scripts, like in Dockerfiles. `apt-get install` has the `-y` option, but it would be useful for all other programs that don't.

It also allows you to script otherwise interactive command line operations with the correct answer. Many come like tools now days provide specific options to override queries. But there are still a couple hold outs which might not.

At that rate no but I definitely use it once in a while. For example if a copy quite a few files and then get repeatedly asked if I want to overwrite the destination (when it's already present). Sure, I could get my commmand back and use the proper flag to "cp" or whatever to overwrite, but it's usually much quicker to just get back the previous line, go at the beginning (C-a), then type "yes | " and be done with it.

Note that you can pass a parameter to "yes" and then it repeats what you passed instead of 'y'.

It's not made to be fast; it's just fast by nature, because there's no other computation it needs to do than to just output the string.

However, my point was just that its performance was never a main point of it. Even without optimizations, it's still very fast, and I don't think whoever created it first was concerned with it having to be super fast, as long as it was faster than the prompts of whatever was downstream in the pipe.

- Node (no buffering): 1.2 MiB/s

- Go (no buffering): 2.4 MiB/s

- Python (8 KiB buffer): 2.7 MiB/s

- Go (8 KiB buffer): 218 MiB/s

Go program:

    f := bufio.NewWriterSize(os.Stdout, 8192)
    for {
       f.WriteRune('1')
    }

More than a few old backup or transfer scripts had extra dd or similar tools in the pipeline to create larger and semi-asynchronous buffers, or to re-size blocks on output to something handled better by the receiver, which was a big deal on high speed tape drives back in the day. I suspect most modern hardware devices have large enough static RAM and fast processors to make that mostly irrelevant.

Rust: 21.9MiB/s

Bash: 282KiB/s

PHP: 2.35MiB/s

Python: 2.30MiB/s

Node: 943KiB/s

In my case, node did not crash after about two minutes. I find it interesting that PHP and Python are comparable for me but not you, but I'm sure there's a plethora of reasons to explain that. I'm not surprised rust is vastly faster and bash vastly slower, I just thought it interesting to compare since I use those languages a lot.

Rust:

  fn main() {
      loop {
          print!("1");
      }
  }

  while :; do printf "1"; done | pv > /dev/null

    $ ./a.out 1000000 2000 | cat >/dev/null
    buffer size: 1000000, num syscalls: 2000, perf:1578.779593 MiB/s
    $ ./a.out 1 2000000 | cat >/dev/null
    buffer size: 1, num syscalls: 2000000, perf:0.832587 MiB/s

    #include <cstddef>
    #include <random>
    #include <chrono>
    #include <cassert>
    #include <array>
    #include <cstdio>
    #include <unistd.h>
    #include <cstring>
    #include <cstdlib>

    int main(int argc, char **argv) {

        int rv;

        assert(argc == 3);
        const unsigned int n = std::atoi(argv[1]);
        char *buf = new char[n];
        std::memset(buf, '1', n);

        const unsigned int k = std::atoi(argv[2]);

        auto start = std::chrono::high_resolution_clock::now();
        for (size_t i = 0; i < k; i++) {
            rv = write(1, buf, n);
            assert(rv == int(n));
        }
        auto stop = std::chrono::high_resolution_clock::now();

        auto duration = stop - start;
        std::chrono::duration<double> secs = duration;

        std::fprintf(stderr, "buffer size: %d, num syscalls: %d, perf:%f MiB/s\n", n, k, (double(n)*k)/(1024*1024)/secs.count());
    }

EDIT 2: Also, it appears that the kernel is smart enough to not copy anything when it's clear that there is no need. When I don't go through cat, I get rates that are well above memory bandwidth, implying that it's not doing any actual work:

    $ ./a.out 1000000 1000 >/dev/null
    buffer size: 1000000, num syscalls: 1000, perf: 1827368.373827 MiB/s

I may be wrong, though. Check with lsof or similar.

https://megous.com/dl/tmp/1046458b5b450018.png

Interestingly, Go is very fast with a 8 KiB buffer (same as Python's), I get 218 MiB/s.

You could probably answer this by replacing printf with /bin/echo and comparing the results. I'm not in front of a Linux box, or I'd try.

[0] https://www.gnu.org/software/bash/manual/html_node/Bash-Buil...

Ah, yeah, good point, I am wrong.

python actually buffers its writes with print only flushing to stdout occasionally, you may want to try:

    python3 -c 'while (1): print (1, end="", flush=True)' | pv > /dev/null

  LuaJIT 2.1.0-beta3

  luajit -e "while true do print('x') end" | pv > /dev/null

  luajit -e "while true do io.write('x') end" | pv > /dev/null

Python cheats, and it's still slow as heck even while cheating (buffers the output at 8192 chunks instead of issuing 1 byte writes).

write(1, "1", 1) loop in C pushes 6.38MiB/s on my PC. :)

It's not language "cheating" of course. It's just OP "measuring the wrong thing".

[0]: https://stackoverflow.com/a/881751/6001364

Using OP's code for following

    php 1.8mb/sec
    python 3.8 Mb/sec
    node 1.0 Mb/sec

    echo 'class Code {public static void main(String[] args) {while (true){System.out.print("1");}}}' >Code.java; javac Code.java ; java Code | pv>/dev/null

    echo 'import java.io.*;class Code2 {public static void main(String[] args) throws IOException {BufferedWriter log = new BufferedWriter(new OutputStreamWriter(System.out));while(true){log.write("1");}}}' > Code2.java ; javac Code2.java ; java Code2 | pv >/dev/null

That manages about 7 GiB/s reusing the same buffer, or about 300 MiB/s with clearing and refilling the buffer every time

(the magic is in using java’s APIs for writing to files/sockets, which are designed for high performance, instead of using the APIs which are designed for writing to stdout)

    node -e '
        const stdoutWrite = util.promisify(process.stdout.write).bind(process.stdout);
        (async () => {
            while (true) {
                await stdoutWrite("1");
            }
        })();
    ' | pv > /dev/null

Perl 5.18.0 gives me 3.5 MiB per second. Perl 5.28.3, 5.30.3, and 5.34.0 gives 4 MiB per second.

    perl5.34.0 -e 'while (){ print 1 }' | pv > /dev/null

    python3 -c 'while (1): print (1)' | pv > /dev/null

    perl5.34.0 -e 'while (){ print "1\n" }' | pv > /dev/null

    perl5.34.0 -E 'while (1) {say 1}' | pv > /dev/null

    package main

    import "fmt"

    func main() {
            for ;; {
                    fmt.Println("1")
            }
    }

Python3: 3 MiB/s

Node: 350 KiB/s

Lua: 12 MiB/s

  lua -e 'while true do io.write("1") end' | pv > /dev/null

  loop = do
    putStr "1"
    loop

  main = loop

  yes | awk '{printf("1")}' | pv > /dev/null

    while true do
      io.write "1"
    end

PUC-Rio 5.4: 25 MiB/s

LuaJIT 2.1.0-beta3: 550 MiB/s <--- WOW

They all go slightly faster if you localize the reference to `io.write`

    local write = io.write
    while true do
      write "1"
    end

No noticeable difference for LuaJIT, which makes sense, since JIT should figure it out without help.

5.1 and 5.4 show about ~8% improvement.

    main = putStr (repeat '1')

Strings are printed one character at a time in Haskell. This choice is justified by unpredictability of the interaction between laziness and buffering; I am uncertain it's the correct choice, but the proper response is to use Text where performance is relevant.

  poll([{fd=1, events=POLLOUT}], 1, 0)    = 1 ([{fd=1, revents=POLLOUT}])
  write(1, "11111111111111111111111111111111"..., 8192) = 8192
  poll([{fd=1, events=POLLOUT}], 1, 0)    = 1 ([{fd=1, revents=POLLOUT}])
  write(1, "11111111111111111111111111111111"..., 8192) = 8192

  poll([{fd=1, events=POLLOUT}], 1, 0)    = 1 ([{fd=1, revents=POLLOUT}])
  write(1, "11111111111111111111111111111111"..., 8192) = 8192
  rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
  clock_gettime(CLOCK_PROCESS_CPUTIME_ID, {tv_sec=0, tv_nsec=920390843}) = 0
  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
  rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
  clock_gettime(CLOCK_PROCESS_CPUTIME_ID, {tv_sec=0, tv_nsec=920666397}) = 0
  ...
  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
  poll([{fd=1, events=POLLOUT}], 1, 0)    = 1 ([{fd=1, revents=POLLOUT}])
  write(1, "11111111111111111111111111111111"..., 8192) = 8192

I'm also not sure what's going on in the second case. IIRC, at some point historically, a sufficiently tight loop could cause trouble with handling SIGINT, so it might be related to some overagressive workaround for that?

Bash:

    while :; do printf "1"; done  | ./pv > /dev/null
    [ 156KiB/s]

    python3 -c 'while (1): print (1, end="")' | ./pv > /dev/null
    [1,02MiB/s]

    perl -e 'while (true) {print 1}'  | ./pv > /dev/null
    [3,03MiB/s]

    node -e 'while (1) process.stdout.write("1");' | ./pv > /dev/null
    [ 482KiB/s]

To do a proper synchronous write, you'd do something like:

  node -e 'const { writeSync } = require("fs"); while (1) writeSync(1, "1");' | pv > /dev/null

Edit: I've found this consistently building multiple data processing applications over multiple years and multiple companies

I'm reminded of this StackOverflow question, Why is reading lines from stdin much slower in C++ than Python?

https://stackoverflow.com/q/9371238/

   python3 -c 'import sys
   while (1): sys.stdout.write("1")'| pv>/dev/null

    python3 -u -c 'import sys
      while (1): sys.stdout.write("1")'| pv>/dev/null

    python3 -c 'import sys
      while (1): sys.stdout.write("1")'| pv>/dev/null

Using python 3.9.7 on macOS Monterey.