The Effect of Pipe Capacity on Unix Pipeline Performance

Re:The Effect of Pipe Capacity on Unix Pipeline Performance by anonsivalley652 on Thursday February 27, 2020 @10:01AM (#22433811)

Awhhh. :'(

I wanted to see:

- What happens when piping an 8 TiB file through awk to be stored on another physical volume (proly want to use xfs).

- A graph of various block sizes, various file sizes, pipeline sizes and timing using a HPET. I don't know how you make an 2+-way pipe like it the diagrams except by process substitution, except that doesn't always work and you can't create cycles without creating FIFO (named pipes):

    cmd0 <(cmd1 here) <(cmd2 here)

    mkfifo -m600 ~joe/foo
    cmd0 <(cmd1 here) <(cmd2 here) | cmd3 > ~joe/foo &
    cmd4 <(cmd5 here) >(cmd6 here ~joe/foo)

    <>(...) # outer can r/w
    <(...) # outer can r
    >(...) # outer can w

[ Parent | on HN ]

Re:The Effect of Pipe Capacity on Unix Pipeline Performance by wnoise on Thursday February 27, 2020 @12:15PM (#22435279)

Most shells don't have great support for making even non-linear graphs of (anonymous) pipe connections (though note that wilder, even dynamic graphs are possible with the "coproc" builtin of e.g. ksh, zsh, or modern bash).

However, arbitrary graphs are possible with the system calls available -- just create all the pipes in the spawning process with pipe(), and after fork(), but before exec() move the right ends into the file descriptors 0, 1, and 2 with dup2().

[ Parent | on HN ]

Re:The Effect of Pipe Capacity on Unix Pipeline Performance by pushedx on Thursday February 27, 2020 @02:18PM (#22436751)

The diagrams are not diagrams of pipelines, they are diagrams of a single pipeline `a | b`, over time, with points in time shown when a is writing and when b is reading.

[ Parent | on HN ]