News for hackers. Stuff that matters.
Re:How are Unix pipes implemented? by dbcurtis on Friday March 27, 2020 @04:24PM (#22706501)
Brian Kernighan gives credit to Doug McIlroy for coming up with the idea in 1964. Quoting from _UNIX A History and a Memoir_, page 68:

"To put my strongest concerns in a nutshell: 1. we should have some way of coupling programs like garden hose..."

from a Bell Labs internal memo.

Kernighan's book is a great read, BTW. Highly recommend to all UNIX nerds.

Re:How are Unix pipes implemented? by yesenadam on Friday March 27, 2020 @07:32PM (#22707852)
Re:How are Unix pipes implemented? by amenonsen on Friday March 27, 2020 @10:00PM (#22708580)
Thank you very much for digging this up, and to all the other commenters here for the useful references.
Re:How are Unix pipes implemented? by tlamponi on Saturday March 28, 2020 @07:41AM (#22710521)
FYI: it's literally in the linked article from this HN post, maybe start reading that, not just the headline and then commenting...
Re:How are Unix pipes implemented? by amenonsen on Saturday March 28, 2020 @07:50AM (#22710552)
FYI: I'm the author of the linked article, and I added a link to the image after it was posted here.
Re:How are Unix pipes implemented? by combatentropy on Saturday March 28, 2020 @04:50PM (#22714222)
This has got to be one of the funniest exchanges I have ever seen on Hacker News.
Re:How are Unix pipes implemented? by tlamponi on Monday March 30, 2020 @01:41AM (#22725538)
Thanks for adding context, that wasn't clear at all to me, it seemed like the classical "Reads headline and then some comments, and post 'smart' stuff which the original article read anyway" response to me.

Sorry for the "accusation" then.

Re:How are Unix pipes implemented? by aasasd on Sunday March 29, 2020 @03:08AM (#22716996)
Now I'm confused and curious as to what the ‘library filing scheme’ means. Does it mean shared libraries? IIRC Unix stuff was statically-linked at first. And then again, what are ‘indexing’ and ‘data path switching’?
Re:How are Unix pipes implemented? by jjice on Friday March 27, 2020 @06:20PM (#22707439)
+1 for Kernighan's book. I'm not one for impulse purchases, but as when I saw a link on HN, I immediately bought it and read through it. I lent it to a professor of mine and he's using as a text for his history course next semester.
Re:How are Unix pipes implemented? by leoc on Friday March 27, 2020 @06:53PM (#22707623)
Here's a nice McIlroy interview with broken formatting: https://www.princeton.edu/~hos/mike/transcripts/mcilroy.htm .
Re:How are Unix pipes implemented? by all2 on Friday March 27, 2020 @04:27PM (#22706538)
Does anyone know how the pipe idea compares to Erlang messaging?

As far as I can tell the implementation of each might look like the other in some ways (a queue of some kind with MutEx guaranteed somehow).

Does anyone have the knowledge to write a little on this? Or maybe point to a similar article for that style of messaging?

Re:How are Unix pipes implemented? by Matthias247 on Friday March 27, 2020 @04:46PM (#22706710)
Pipes are bytes Streams. Communication between actors as well as CSP channels are object streams.

Pipes exercise backpressure on the writer - if the pipe is full the writer is blocked. Actor systems mostly use seemingly unbounded queues. The sender will not get blocked.

Re:How are Unix pipes implemented? by toast0 on Friday March 27, 2020 @09:05PM (#22708330)
> Pipes exercise backpressure on the writer - if the pipe is full the writer is blocked. Actor systems mostly use seemingly unbounded queues. The sender will not get blocked.

Erlang does suspend (block) processes that send to ports or nodes when the buffers for that get full; but not when sending to local processes. There used to be an optional reduction count punishment for senders when sending to messages with larger mailboxes, but it seems that may have been removed. I don't think it would be too hard to add a feature where sending to a local mailbox over a specified size caused the sender to be suspended, but tracking it might be a little difficult.

Re:How are Unix pipes implemented? by wahern on Friday March 27, 2020 @05:03PM (#22706876)
Here's a short 2014 paper from McIlroy himself discussing his early thinking back then: https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf

TL;DR: McIlroy was applying the concept of coroutines, which was described by Melvin Conway in 1963. Two processes communicating over a pipe are basically coroutines, except instead of passing structured data they're just passing bytes.

A Unix pipeline is a set of multiple coroutines. In fact, Tony Hoare's 1978 Communicating Sequential Processes paper cites the UNIX shell[1] for the concept of coroutines, and discussion of coroutines figures prominently in that paper. See https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf CSP basically models the behavior of a large set of coroutines.

AFAIU, Erlang was partly inspired by CSP. You can draw a straight line from coroutines, through Unix pipelines and CSP, to Erlang's processes.

[1] Specifically, Ken Thompson's 1976 paper, "The UNIX command language." See https://archive.org/details/theunixcommandlanguage

Re:How are Unix pipes implemented? by TheGrassyKnoll on Saturday March 28, 2020 @04:54AM (#22710013)
"...a practical exploration of using Python coroutines..."

  http://www.dabeaz.com/coroutines/index.html