Knuth really is a fan of writing monolithic (rather than "modular") programs from scratch, in a way that goes against all the experience of software engineering accumulated over decades, so that criticism is well-deserved.
For example, his big programs TeX (1982) and METAFONT (1984) are each book-length and the source code of each is in a single large file amounting to about 20000+ lines of Pascal code. His programs do not contain much in the way of standard software-engineering practices like abstraction, modules (hiding implementation behind an interface), unit tests, libraries, etc. In fact, he has spoken out against unit tests and code reuse! [1]
> the idea of immediate compilation and "unit tests" appeals to me only rarely, when I’m feeling my way in a totally unknown environment and need feedback about what works and what doesn’t. Otherwise, lots of time is wasted on activities that I simply never need to perform or even think about. Nothing needs to be "mocked up." ...
> With the caveat that there’s no reason anybody should care about the opinions of a computer scientist/mathematician like me regarding software development, [...] I also must confess to a strong bias against the fashion for reusable code. To me, "re-editable code" is much, much better than an untouchable black box or toolkit. I could go on and on about this. If you’re totally convinced that reusable code is wonderful, I probably won’t be able to sway you anyway, but you’ll never convince me that reusable code isn’t mostly a menace.
Moreover, his sympathies always lay with the "other" side of the "structured programming" revolution (he still liberally uses GOTOs, etc -- still coding like a 1950s/1960s machine code programmer), and in his 1974 paper "Structured Programming With Go To Statements", he approvingly quotes something that might horrify many software engineers today:
> In this regard I would like to quote some observations made recently by Pierre-Arnoul de Marneffe:
> In civil engineering design, it is presently a mandatory concept known as the "Shanley Design Criterion" to collect several functions into one part . . . If you make a cross-section of, for instance, the German V-2, you find external skin, structural rods, tank wall, etc. If you cut across the Saturn-B moon rocket, you find only an external skin which is at the same time a structural component and the tank wall. Rocketry engineers have used the "Shanley Principle" thoroughly when they use the fuel pressure inside the tank to improve the rigidity of the external skin! . . . People can argue that structured programs, even if they work correctly, will look like laboratory prototypes where you can discern all the individual components, but which are not daily usable. Building "integrated" products is an engineering principle as valuable as structuring the design process.
> ... Engineering has two phases, structuring and integration: we ought not to forget either one...
(This comment is slightly tongue-in-cheek, but hopefully provocative enough.)
[0]: Hey it's been a couple of hours and there's no reply attacking my comment, guess I better do it myself. :-)
[1]: http://www.informit.com/articles/article.aspx?p=1193856
Knowing that the one who is possibly the most respected person in the field of computer science has an opinion that goes against the current trends gives us perspective.
In the same way, I don't fully agree with Richard Stallman's activism and Linus Torvald's famous rants, but I'm glad there are people like that to shake things up.
Computer science is not software engineering. You also don't get physicists and mathematicians determine engineering best practices for precisely the same reason.
For example, the reference to Shanley's design principle is not only wrong in it's core (aerospace engineering instead of civil engineering) it also completely misses the underlying design requirements and constraints. More precisely, aerospace pays a premium for weight, thus design requirements emphasize the need to optimize structural systems wrt structural weight. This design choice favours operational economy at the expense of production costs, maintainability, robustness, and simplifying analysis.
None of this applies to software development, or even civil engineering structures.
Appeals to authority are a dangerous fallacy.
But generally, algorithms researchers seem to not care about abstractions, as witnessed by TeX and LaTeX in multiple ways.
That's probably because when you really need to invent a fancy algorithm (which is their job), that will often not be built out of reusable components.
You need to be careful not to confuse experience and common practice with empirically proven benefit. Many of the practices that you mention are intended to increase the feasibility (perceived feasibility!) of industrial reuse, and/or division of labor, not to make software better in any other dimension (reliability, code size, power/time/space efficiency, fitness for purpose.)
Actually, nowadays the main driving force for abstractions and modularity is to make the code easier to test, understand, and refactor.
Furthermore, "power/time/space efficiency, fitness for purpose" are concerns that don't really apply to software development. In general the main resource is labour, and all other requirements are secondary (computational reaources, latency, etc)
Part of the problem is this almost hero-worshipping nature of underlying libs we rely on; and part of the problem is one of perception: even in today's reuse-fetishized culture, likely almost all code is of the non-reused kind; yet in any given program we see huge amounts of reused code imported from e.g. package managers - because those reused bits are often reused a lot.
We'd be much better at reuse if we were a little more sskeptical of it, and didn't assume that design rules that hold for code that's been packaged for reuse also hold for the more pedestrian but nevertheless very common code that is not currently being reused. We want strict, leak-free abstractions ideally covering both functional and non-functional aspects for the reusable bits; but where we cannot do that or cannot yet afford to, we want the opposite: better clearly transparent code than a mess of leaky abstractions.
By the same token we don't copy and paste code enough. Sometimes a good abstraction is elusive, yet a pattern is still recognizable and useful. We have language features and a culture surrounding directly reusable code, but no such habits for derivative code, even though that would be quite useful. Essentially: I'm perfectly happy to deal with people using some stackoverflow answer to write code, but as soon as people do, it's like we regress into the dark ages: there's no structured citation, no support for detecting updates, no "package manager" that tracks updates (for stackoverflow clearly doesn't do that), and no diff or whatever to show how you tweaked the code segment. So instead people all too often just throw future maintainers under the bus with some random code-golfed answer, or some "reused" library that is hardly much easier to use than the reimplementation, with much harder to spot gotchas and perf/security issues, and often an API that isn't actually convenient for your use case.
So yeah: software needs reusable components, but 99% of the code you write should be re-editable, not reusable; and 100% of the time you should aim for re-editability, and only ever grudgingly accept the need for reusability after multiple use-cases are found (not just 2 or 3!), and there is a leak-free abstraction possible, and you've considered things like perf and debuggability and security.
That could be interpreted in non-horrifying ways. Sure, the cost of a function call (especially with a modern compiler that can intelligently inline) is negligible in most cases compared with the engineering benefits it offers.
But what about cases where it's less clear? In databases, there are constantly cycles between:
Cycle phase 0: Abstract Storage: separate storage from computation to make the database system easier to test, replace, and reconfigure.
Cycle phase 1: Push computation down to storage: wow, look how much better performance we can get if we intermingle storage and computation!
GOTO cycle phase 0.
You seem to be claiming that 20,000 lines of code constitutes a large program. Is that really your intention? I mean libjpeg is 34,000 lines of code, and LAPACK 3.6.0 (one of the very few libraries that excels TeX in reusability and enduring value) is 685,000 lines of code, and each of them is just a small part of many programs. I would instead describe the monolithic parts of TeX and METAFONT as small programs of only 20,000 lines of code, omitting even dependencies on libraries.
(Ironically, in my previous job supposedly using "modern" programming practices, a single Python file had organically grown to over 25000 lines in length and people complained to GitHub about the file not being rendered in full in the browser.)
Incidentally you got it completely backwards. TeX is used because it's a convenient interface between higher level descriptions (i.e., book content) and the lower level output (pretty document formats). Thus, once again abstractions and interfaces show their value.
Additionally, TeX is used rarely by humans, while LaTeX is the tried and true workhorse. Again, an abstraction that targets a interface.
And how many TeX and LaTeX reimplementation a are there? Again, the interface and abstractions show their value.
This is not just a "poor me, I am being persecuted" issue. We can let Hacker News turn into Twitter, with insult contests being resolved by flagging campaigns that eventually hellban the accounts of the less-popular side of any issue, or into YouTube, dominated by conspiracy theories and hate; or we can stand up for reasoned discussion and informed comment.
Also, note that the person who posted the misinformation never bothered to thank svat for their careful and courteous correction and their meticulous attempt to try to DWIM the troll comment into something that made some amount of sense. This makes more sense on the assumption that they were trolling than on the assumption that they were merely misinformed — someone who was interested in the truth would surely have offered thanks for such a helpful and polite correction. Instead, it seems clear that they were only posting here to make trouble and waste the time of reasonable, knowledgeable people like svat — particularly in light of https://news.ycombinator.com/item?id=22424205.
I had already come to that conclusion because the comment was clearly based on a set of misconceptions about what TeX and LaTeX are, and how they relate to each other, which could have been corrected by reading the introductory paragraphs of the Wikipedia article for either TeX or LaTeX. If someone isn't investing even that level of care in their comments, they don't care whether they're talking nonsense or not.
We can see that the same account continues to post aggressive comments rudely attacking other users, although since I'm not familiar with the areas they're talking about, I can't tell if they're employing the same reckless disregard for the truth in these cases: https://news.ycombinator.com/item?id=22501626 "I don't see the point of your post, and frankly sounds like nitpicking."
https://news.ycombinator.com/item?id=22499637 "Don't you understand where and why are there abstractions? ... having people [referring to the person he's replying to] naively complain"
https://news.ycombinator.com/item?id=22458752 "You may quote simplistic cliches"
https://news.ycombinator.com/item?id=22459638 "Your comment is unrealistic and even absurd"
Maybe you think this is the kind of dialogue we should be encouraging on here, but I don't. I think that in addition to setting an example of better behavior, as svat did, we should explicitly call out such misbehavior and explain why it isn't desirable, as I did.
> TeX is used because it's a convenient interface between higher level descriptions…
I think you mean LaTeX, not TeX. (LaTeX is a macro layer on top of TeX that provides these convenient interfaces, while TeX is a low-level typesetter.)
> Additionally, TeX is used rarely by humans…
This sounds a bit contradictory with the previous statement, but maybe you mean that TeX is rarely used directly by many people (without LaTeX or some other macro layer). In any case, the LaTeX macros are implemented in TeX, so the TeX program is always the one being used (which I think was the point of the poster you're replying to).
> And how many TeX and LaTeX reimplementation a are there?
I didn't understand the meaning or point of this, as the answer is either "very few" or "many" depending on what you're counting. Extensions of the TeX program include eTeX, pdfTeX, XeTeX and LuaTeX, not to mention a few others like pTeX and upTeX. (Confusingly, when we say “TeX” we often mean one of these programs as well, as a lot of their code comes from TeX—they are written/implemented as patches (changefiles).) Reimplementations of a small part of the TeX/LaTeX syntax for mathematical expressions (only) include MathJax and KaTeX. Are these a lot, or hardly any (would have expected a lot more)? Depends on your perspective I guess.
It's probably worth pointing out that nobody uses TeX bare without a LaTeX-like macro library. Knuth wrote his books with a macro library confusingly called "plain TeX", which ships with the TeX language interpreter. (That interpreter was the subject of this thread.) LaTeX relies on the plain TeX library, just as, for example, GLib relies on the C standard library. The third alternative popular TeX macro library for document formatting, other than plain TeX and LaTeX, is a thing called ConTeXt. It seems to me that ConTeXt is less popular than LaTeX, but more popular than plain TeX.
But of course to invoke any of these you have to write code in TeX, just as to invoke Rails you have to write code in Ruby, or to invoke Numpy you have to write code in Python.
As far as reimplementations, I think the only reimplementations were done at Stanford in the late 1970s as Knuth and his students wrote a series of prototypes, culminating in the TeX language we know today in 1983.
Well said.
Hiding complexity is harder than handling it. Designing an effective way to expose it for use amounts to (part of) a "programming system" [Brooks].
So it's library writers who hide the complexity for average programmers, with a "programming product": stdlib, open source project or (rarer these days) commercial "engine".
When there is no such library, we get the current state of our art: many projects failing.
The point of literate programming is that you have a text which is structured in a way that humans are good at dealing with, which is not some nifty mathematical abstraction like functional or oop code nor coherent modules.
(Having worked with a badly structured monolith with a mixture of literal code and a million bad abstractions, I must admit the simplicity of working with the used-once literal code has its advantages. But so does the more abstract code. The only thing I know is that I want to control my entry point rather than go via a framework and suffer my concrete classes getting upcasted.)
He also used six-character identifiers. He had to. TeX and Metafont were intended to be compiled by lowest common denominator Pascal compilers. Modules, etc., were all vendor specific extensions at the time.
One might consider that to be good engineering.
"If you cut across the Saturn-B moon rocket, you find only an external skin which is at the same time a structural component and the tank wall."
Soda cans and plastic water bottles are the same; it's how they are cheap enough to be fit for for purpose.
I've had to check myself and indeed we have different ideas about the meaning of "speaking against unit tests".
Did you know about all the tests he wrote and run for TEX? There's a book about it as well.
By "unit tests" Knuth is speaking of tests that each test just one part of the program in isolation, mocking out the rest, and also more broadly about the practice of "test-driven development" which has this as its prerequisite. He means he finds it simpler to just write the whole program up front, and test it wholesale.
Even in real hard engineering fields we see teams responsible for specific subsystems and components, which they need to test and validate.
I think the key takeaway here is that it's better all around if somebody like Knuth spends their time and effort on those Unix programs - or, in general, the reusable blocks - and leaves putting them together in various ways to solve the different issues at hand to the less qualified.
What Knuth seems to enjoy are ideas, and teaching/exposition. That might explain why his "literate programming" is structured around the metaphor of writing, rather than that of, say, engineering, architecture, or tool-making. In this case, he has written an “essay” around a certain data structure that we may call a hash-packed trie, a small twist (possibly never done before!) on the packed tries that had been described in his student Frank Liang's thesis and used for hyphenation in the TeX program.
This data structure is uniquely suited to this problem of going through a file and counting words. It cannot be obtained by putting together "reusable blocks". (At most it can itself be packaged as a library, but would probably be used only for word-counting, which is the original problem itself.) So I would disagree with your "key takeaway" IMO. What you (and McIlroy) are describing is a different approach where one does not try to arrange for a problem to be solved using the data structures and algorithms best suited for it, but tries to express the problem in terms of existing “blocks” even at some cost of efficiency or other considerations. This is a perfectly fine philosophy on its own (see the quote above about "prefabricated house"), but it's not the full story here.
Given Knuth's background, it may well be that he was looking for the (computationally) optimal solution, wherein the bash script was more about writing it quickly using existing tools.
1. Note that these were not quite "two solutions" as one might think from the phrase, but rather a commissioned program, and a review of it (which also happened to contain an alternative approach).
2. I did actually end up comparing them on a test corpus just last month -- kind of linked to it twice in the comment but here it is again: https://codegolf.stackexchange.com/questions/188133/bentleys... -- a C++ translation of Knuth's program runs about 140 times faster on the largest case than the shell solution.
3. About "writing it quickly using existing tools": note that while the Unix commands -- tr, sort, uniq, sed -- are widely available today (even on Windows!), they were not so in 1986, unless you were in certain special circumstances such as having paid a lot of money for a Unix system. So McIlroy was pointing out that in their (Unix) world, this problem would be easier than it was outside, so his proposed solution was also sort of an advertisement for Unix, not just the Unix philosophy.
https://news.ycombinator.com/item?id=22221592
https://news.ycombinator.com/item?id=18699718
For further context you can look at past and future issues of Bentley's column (and its spinoff); a list of them I collected here: https://shreevatsa.net/post/programming-pearls/
I guess it's a long-standing tradition in literary reviews for reviewers to push their own ideas, rather than confining themselves solely to reviewing the work in question. That is what happened here. Knuth had written a program that he had been asked to write, to demonstrate the programming discipline. But McIlroy, as the inventor of Unix pipes and a representative of the Unix philosophy (at that time not well-known outside the few Unix strongholds: Bell Labs, Berkeley, etc), decided to point out (in addition to a good review of the program itself) the Unix idea that such special-purpose programs shouldn't be written in the first place; instead one must first accumulate a bunch of useful programs (such as those provided by Unix), with ways of composing them (such as Unix pipes). A while later, John Gilbert described this episode this way:
> Architecture may be a better metaphor than writing for an endeavor that closely mixes art, science, craft, and engineering. “Put up a house on a creek near a waterfall,” we say, and look at what each artisan does: The artist, Frank Lloyd Wright (or Don Knuth), designs Fallingwater, a building beautiful within its setting, comfortable as a dwelling, audacious in technique, and masterful in execution. Doug McIlroy, consummate engineer, disdains to practice architecture at all on such a pedestrian task; he hauls in the pieces of a prefabricated house and has the roof up that afternoon. (After all, his firm makes the best prefabs in the business.)
There are other points (not mentioned in this article), e.g. the fact that someone had to have written those Unix programs in the first place and writing them with literate programming can lead to better results, and the fact that Knuth's idea of using a trie (though not a packed/hash trie; that's no longer needed) still seems fastest: https://codegolf.stackexchange.com/questions/188133/bentleys... (please someone prove me wrong; I'd love to learn!)
Knuth gladly included McIlroy's review verbatim when he reprinted this paper in his collection Literate Programming. BTW here's an 1989 interview of McIlroy https://www.princeton.edu/~hos/mike/transcripts/mcilroy.htm where he looks back and calls Knuth's WEB “a beautiful idea” and “Really elegant”, and his review “a little unfair”, though of course he reiterates his main point.