Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by jws on Saturday February 21, 2015 @10:13PM
(#9088046)
Looks like Plan 9 and FreeBSD are the only ones to correctly handle write errors. The others just silently ignore them. I guess the moral is "Don't rely on echo to save important data."
Also, the comment…
/* This utility may NOT do getopt(3) option parsing. */
… which appears in two of them certainly cries out for an explanation. Was it true in NetBSD? Is it still true? Will people call us names if we do?
Update: bash does not handle errors. zsh might, it checks the result of fclose() or if stdout, fflush(). I'm not sure that is sufficient after ignoring all the fwrite() and fputc() errors. Plus it's in an 850 line function and my monitor is only 20" tall, so I'm not following all of it.
Update2: asveikau is correct. The only ones to attempt error handling miss short reads. FreeBSD also misses EAGAIN/EINTR handling. Plan9 doesn't need that, but its man page explicitly states that short writes should be considered errors by the caller.
I guess it's official. The unix read/write system calls are too complicated to be used by experts in the operating system. (I've rather thought that for some time now.)
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by anon1385 on Sunday February 22, 2015 @02:42AM
(#9088636)
There was actually a project for the GNU Coreutils to make sure that write errors were being reported[1]. It looks like echo is using the "atexit (close_stdout);" method described in that talk to close the stream and detect errors. You can see what close_stdout actually does here: http://git.savannah.gnu.org/cgit/gnulib.git/tree/lib/closeou...
Arguably GNU echo is the only one to actually get this correct. Which is kinda amusing given how the comments on github are all about how ugly and bloated it is and how elegant the Plan9 and UnixV ones are.
The talk is worth a watch if you care about these things. It kind of re-enforces your point about read/write being to complicated to use correctly. Meyering found these kinds of bugs in lots of other programs like perl, python, rsync and emacs.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by asveikau on Saturday February 21, 2015 @10:22PM
(#9088065)
> Looks like Plan 9 and FreeBSD are the only ones to correctly handle write errors.
Both only check for negative return codes. If stdout were piped/redirected to something where write(2) may return a short byte count (let's say a socket) neither of them will detect it.
> /* This utility may NOT do getopt(3) option parsing. */
I noticed this too. My guess, totally not backed up by anything but by instinct, is getopt(3) tried to do too much which doesn't make sense for echo. For example if it tried to parse "echo -foobar" as anything other than "you spit out the string '-foobar'" I could see it being problematic.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by jws on Saturday February 21, 2015 @11:10PM
(#9088193)
So POSIX allows no options to the command, but if the first operand is "-n" then the result is undefined. that compromise between the BSD and SysV behavior just have taken some time to work out.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by jcoffland on Saturday February 21, 2015 @10:33PM
(#9088098)
There is nothing wrong with complex code that is correct and readable. I imagine those who have a problem with the GNU version have not done a lot of shell scripting and therefore don't see a need for the complexity. The system V version would really suck in a lot of situations. Of course people love to criticize because it makes it sound like they know something.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by angersock on Sunday February 22, 2015 @12:07AM
(#9088312)
Generally no, other than the fact that it is complex--complexity itself is a sin. Always has been, always will be, though it can be forgiven.
Anyways, what's the issue with the System V version, other than not handling newlines?
Also, what is the purpose with the Plan 9 loading the thing into a buffer first, other than perhaps to get the benefit of an all-or-nothing write? I suspect weirdness with the Plan 9 file handling but am not certain.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by sjolsen on Sunday February 22, 2015 @11:36PM
(#9092568)
>complexity itself is a sin
Complexity is inherent to any task in some degree. That complexity must be expressed in full in any program which performs that task correctly, whether in terms of language/library constructions or the ways in which those are combined. When the tools with which you implement a task (here, C and the core Unix API) are structured substantially differently than the high-level concepts of which that task is composed, there is additional complexity inherent to mapping those concepts.
In light of that, I don't understand what you mean by, "complexity itself is a sin," unless you're referring specifically to the strictly unnecessary complexity introduced by a sub-optimal mapping of high-level semantics to core constructs. I don't think that's the kind of complexity we're seeing here. I think we're seeing the complexity inherent to mapping shell-level semantics to C semantics and the semantics of the Unix API, specifically error-handling semantics.
I agree with jws that what this code demonstrates is that the core Unix API is difficult to use for writing precisely correct high-level programs, even ones as simple as echo. I feel that this is exactly why it's important that core high-level programs like echo encapsulate as much of the complexity of mapping to C as possible (that is, in the style of GNU rather than SysV): so that one can easily write reliable programs with high-level tools like the shell without worrying about that complexity.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by f2f on Saturday February 21, 2015 @10:33PM
(#9088099)
what this fine list sorely misses is a true retelling of the "Plan 9 and the echo" story, as found here, on the Plan 9 mailing lists some millenia ago:
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by BuildTheRobots on Saturday February 21, 2015 @11:02PM
(#9088172)
Excellent link; thank you.
It references _The Unix Programming Environment_, Kernighan & Pike (1984), pp. 77-79, "A digression on echo" which I found in PDF and is an immensely enjoyable and enlightening read :)
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by ori_b on Sunday February 22, 2015 @01:32AM
(#9088502)
Note that the plan9 version does some extra work so that it can send the entire buffer in one write() operation. This is because Plan 9 guarantees that messages smaller than the size of a pipe buffer are sent as an atomic write, given a sufficiently large recieve buffer on the other end.
This simplifies quite a bit of code that reads and writes commands into ctl files.
I wonder if, like a "Hello world" program, some of these simpler utilities might not even meet the minimum level of creativity to be eligible for copyright, so complexity was added - true.c and false.c are the other two examples I can think of; the "true" utility is basically generated by the default code template provided by many IDEs.
There's also a fun story floating around called "The UNIX and the Echo" about what it should do without any arguments, or whether or not it should accept options or escape sequences:
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by tbirdz on Saturday February 21, 2015 @09:29PM
(#9087925)
It would be more worthwhile to show various shells implementations of echo, because most of the time you are going to be running your shell's builtin echo instead of /bin/echo.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by Zenst on Saturday February 21, 2015 @08:59PM
(#9087861)
Wonderful to see the various flavours of the same simple command. Though GNU using a goto for what at quick glance is a non error situation does not sit well with me, but probably personal taste in code.
ALso surprised how many library's the FreeBSD and GNU flavours want to pull in.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by stock_toaster on Saturday February 21, 2015 @09:54PM
(#9088006)
Looking at the history of echo.c in the freebsd tree[1] is pretty interesting too. Seems before this [2] commit (apparently fixes for performance with many arguments) it was pretty simple.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by userbinator on Sunday February 22, 2015 @01:02AM
(#9088441)
To me this feels like a classic case of "premature optimisation" - by increasing the complexity of the code quite a bit (even adding dynamic allocation), they managed to save only 40 milliseconds on a benchmark involving the echoing of 1000 arguments.
Using 1000 arguments does not seem at all a typical use case for echo.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by jlarocco on Sunday February 22, 2015 @01:50AM
(#9088544)
I'm not sure "premature" is the right word. I mean, they benchmarked, and maybe even profiled, and made it faster.
My guess is that somone had a weird use case for echo where the old version was slow. He benchmarked, profiled, made some changes, and submitted a patch. "The source code is too long," is a crappy argument, so might as well use the faster, longer code.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by ild on Sunday February 22, 2015 @01:28PM
(#9090351)
Well 40 ms can be a lot or a little, depending on the number after optimization. If, for example the input was 1000 1-letter arguments, 40 ms to output 2kb is too much.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by muyuu on Saturday February 21, 2015 @09:51PM
(#9088000)
Hehe I went in fully expecting GNU to have managed making something as simple as echo.c relatively complex and with inconceivable options. Was not disappointed. They probably even managed to include a GNUism or two.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by caf on Sunday February 22, 2015 @05:55AM
(#9089020)
The GNU echo is designed to be useable on a variety of systems which had different traditional behaviours - this comment in the source code is instructive:
/* System V machines already have a /bin/sh with a v9 behavior.
Use the identical behavior for these machines so that the
existing system shell scripts won't barf. */
(BSD systems supported suppressing newline with -n; UNIX System V systems supported \ escape sequences; POSIX came later and allowed either unless the XSI option is supported).
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by muyuu on Sunday February 22, 2015 @07:22AM
(#9089141)
Yep, including all that hydra-style-compatibility as part of the very base implementation of anything, is part of what makes GNU code the way it is.
I mean... in main() there's a labeled goto for "just_echo" - even that section alone has something like 6 or 7 indentation levels. If I was writing satire code of GNU style I'd probably wouldn't have gone so far. Hehehe.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by masklinn on Sunday February 22, 2015 @08:10AM
(#9089253)
Then again, so do the BSD tools. And of course the options are often not compatible between the two e.g. bsd sed's -l enables line buffering, gnu sed's -l specifies line-wrapping length; bsd sed's -i requires an extension (empty for no backup) while gnu sed's does not.
POSIX sed supports exactly 3 options (-n, -e and -f), the latest FreeBSD sed supports 10 (adding -E, -a, -I, -i, -l, -r and -u — the last two being GNUism compatibility options not necessarily available on older versions, they are not on my OSX machine) and GNU sed supports 9 short and an additional 4 long options. And that's not counting the extensions to the sed command set.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by pjmlp on Sunday February 22, 2015 @10:35AM
(#9089703)
True, which is why it is hard to keep up with multiple UNIX implementations and to know what is supported on a given box without having to reach for man first.
OPERANDS
The following operands shall be supported:
string
A string to be written to standard output.
[...]
The following character sequences shall be recognized on XSI-conformant systems within any of the arguments:
[...]
\c
Suppress the <newline> that otherwise follows the final argument in the output. All characters following the '\c' in the arguments shall be ignored.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by cogburnd02 on Sunday February 22, 2015 @07:03AM
(#9089100)
Well, if you're going to downvote me, at least explain why you disagree; don't hide behind the mouse-click. I was merely asking why another of the freely available versions of echo was missing from this list; what's wrong with that?
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by Jach on Sunday February 22, 2015 @08:39AM
(#9089352)
Are you honestly expecting the author of this 4 year old commit to respond? Maybe if you included a link to the other version (even though it's already in a github comment) your first comment would have been useful and not downvoted, but as it is it's just pointlessly shared wondering that if serious would be better done as a private message to the author... as for your second, complaining about downvotes generally results in more.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by Jach on Sunday February 22, 2015 @09:47PM
(#9092295)
I downvoted the post complaining about downvotes, but not the first post, since it was already downvoted. (Few things need to be downvoted/flagged to oblivion.) Since my attempt at explaining why the first post was downvoted received upvotes, at least we can surmise it's not totally off the mark to what others may think.
Re:Unix V5, OpenBSD, Plan 9, FreeBSD, and GNU Coreutils Impl...
by unwind on Sunday February 22, 2015 @07:57AM
(#9089219)
I wondered the same thing, or rather was going to post a link to busybox here, but it's already in a comment to the original article. Not sure that's why you got downvoted though, seems a bit harsh as re-posting interesting links related to the original post is quite common here and not something I thought was frowned upon.
Also, the comment…
… which appears in two of them certainly cries out for an explanation. Was it true in NetBSD? Is it still true? Will people call us names if we do?Update: bash does not handle errors. zsh might, it checks the result of fclose() or if stdout, fflush(). I'm not sure that is sufficient after ignoring all the fwrite() and fputc() errors. Plus it's in an 850 line function and my monitor is only 20" tall, so I'm not following all of it.
Update2: asveikau is correct. The only ones to attempt error handling miss short reads. FreeBSD also misses EAGAIN/EINTR handling. Plan9 doesn't need that, but its man page explicitly states that short writes should be considered errors by the caller.
I guess it's official. The unix read/write system calls are too complicated to be used by experts in the operating system. (I've rather thought that for some time now.)
Arguably GNU echo is the only one to actually get this correct. Which is kinda amusing given how the comments on github are all about how ugly and bloated it is and how elegant the Plan9 and UnixV ones are.
The talk is worth a watch if you care about these things. It kind of re-enforces your point about read/write being to complicated to use correctly. Meyering found these kinds of bugs in lots of other programs like perl, python, rsync and emacs.
[1] http://www.irill.org/events/ghm-gnu-hackers-meeting/videos/j... (slides: http://www.gnu.org/ghm/2011/paris/slides/jim-meyering-goodby... )
Both only check for negative return codes. If stdout were piped/redirected to something where write(2) may return a short byte count (let's say a socket) neither of them will detect it.
> /* This utility may NOT do getopt(3) option parsing. */
I noticed this too. My guess, totally not backed up by anything but by instinct, is getopt(3) tried to do too much which doesn't make sense for echo. For example if it tried to parse "echo -foobar" as anything other than "you spit out the string '-foobar'" I could see it being problematic.
Anyways, what's the issue with the System V version, other than not handling newlines?
Also, what is the purpose with the Plan 9 loading the thing into a buffer first, other than perhaps to get the benefit of an all-or-nothing write? I suspect weirdness with the Plan 9 file handling but am not certain.
Complexity is inherent to any task in some degree. That complexity must be expressed in full in any program which performs that task correctly, whether in terms of language/library constructions or the ways in which those are combined. When the tools with which you implement a task (here, C and the core Unix API) are structured substantially differently than the high-level concepts of which that task is composed, there is additional complexity inherent to mapping those concepts.
In light of that, I don't understand what you mean by, "complexity itself is a sin," unless you're referring specifically to the strictly unnecessary complexity introduced by a sub-optimal mapping of high-level semantics to core constructs. I don't think that's the kind of complexity we're seeing here. I think we're seeing the complexity inherent to mapping shell-level semantics to C semantics and the semantics of the Unix API, specifically error-handling semantics.
I agree with jws that what this code demonstrates is that the core Unix API is difficult to use for writing precisely correct high-level programs, even ones as simple as echo. I feel that this is exactly why it's important that core high-level programs like echo encapsulate as much of the complexity of mapping to C as possible (that is, in the style of GNU rather than SysV): so that one can easily write reliable programs with high-level tools like the shell without worrying about that complexity.
The Plan 9 and the Echo:
http://9fans.net/archive/2001/09/54
(for the history buffs: yes, that is dennis ritchie posting in that thread)
It references _The Unix Programming Environment_, Kernighan & Pike (1984), pp. 77-79, "A digression on echo" which I found in PDF and is an immensely enjoyable and enlightening read :)
[1] http://books.cat-v.org/computer-science/unix-programming-env...
This simplifies quite a bit of code that reads and writes commands into ctl files.
In other words, writev on plan9 isn't a system call -- it does the same copying that echo.c does. Also, echo.c probably predates writev.c.
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/bin/echo/echo.c...
http://en.wikipedia.org/wiki/Echo_(command)
I wonder if, like a "Hello world" program, some of these simpler utilities might not even meet the minimum level of creativity to be eligible for copyright, so complexity was added - true.c and false.c are the other two examples I can think of; the "true" utility is basically generated by the default code template provided by many IDEs.
There's also a fun story floating around called "The UNIX and the Echo" about what it should do without any arguments, or whether or not it should accept options or escape sequences:
http://stackoverflow.com/questions/3290683/bloated-echo-comm...
The official POSIX definition leaves it implementation-defined, with XSI forbidding options and requiring escape sequences:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ec...
It's amazing how much variety can be present for functionality that seems so simple at first glance.
ZSH: http://sourceforge.net/p/zsh/code/ci/a4ff8e69570cbdb8e7d5bf1...
ALso surprised how many library's the FreeBSD and GNU flavours want to pull in.
[1]: https://github.com/freebsd/freebsd/blob/master/bin/echo/echo...
[2]: https://github.com/freebsd/freebsd/commit/cbf5708f4336b824b8...
Using 1000 arguments does not seem at all a typical use case for echo.
My guess is that somone had a weird use case for echo where the old version was slow. He benchmarked, profiled, made some changes, and submitted a patch. "The source code is too long," is a crappy argument, so might as well use the faster, longer code.
https://github.com/uutils/coreutils/blob/master/src/echo/ech...
[0] http://git.suckless.org/sbase/tree/echo.c
I mean... in main() there's a labeled goto for "just_echo" - even that section alone has something like 6 or 7 indentation levels. If I was writing satire code of GNU style I'd probably wouldn't have gone so far. Hehehe.
POSIX sed supports exactly 3 options (-n, -e and -f), the latest FreeBSD sed supports 10 (adding -E, -a, -I, -i, -l, -r and -u — the last two being GNUism compatibility options not necessarily available on older versions, they are not on my OSX machine) and GNU sed supports 9 short and an additional 4 long options. And that's not counting the extensions to the sed command set.
Definitely less convenient, but would make building cross-platform scripts much easier.
http://www.meetup.com/Classical-Code-Reading-Group-of-New-Yo...
I know those guys were pretty smart, so I'm legitimately curious.
i==argc? '\n': ' '
part of the print statement cleaner.
I meant --no-nl or similar.