Any recommendations/suggestions?
1) Foundations of Multithreaded, Parallel, and Distributed Programming by Gregory R. Andrews - This is one of my favourites even though old. You get to learn/compare the different paradigms in one single book.
2) Parallel Programming: Concepts and Practice by Bertil Schmidt et al. - This is a more recent book with good explanations/coverage including CUDA.
3) The Art of Multiprocessor Programming by Maurice Herlihy et al. - Well known classic though with a main focus on shared-memory architectures.
Along with the above you also need more detailed language/library specific implementation books;
a) C++ Concurrency in Action by Anthony Williams.
b) Programming with POSIX Threads by David Butenhof.
c) UNIX Systems Programming: Communication, Concurrency and Threads by Kay Robbins and Steve Robbins.
Finally; for studying concurrency at the OS level where software meets hardware, i have found nothing better than;
i) Unix Systems for Modern Architectures: Symmetric Multiprocessing and Caching for Kernel Programmers by Curt Schimmel.
In 2015, I decided I was no longer poor and should buy the box-set as a treat. I set aside time to read a chapter in the first week after it arrived, but realised that it would take years to read and actually understand the material in that one book, let alone the other two. I decided I'd just dig in from time to time, but found that I never actually did, because in all honestly I'm less interested in abstract algorithms now, and prefer to spend free time doing other things in life that don't involve computers.
So, my copy is still essentially untouched after almost a decade.
Also, it's really heavy (I'd say at least over 1kg), so by the time I sold it at less than the current price because it's no longer new (also mine is 1-4A, the current edition is 1-4B) and paid postage to package and ship it somewhere, it's probably not really worth the effort.
You should imagine that you are forced to sell it every day. Eventually you'll either sell it (or donate it (your local library might appreciate it)) or start reading it.
Semi-serious question: Which of the following do you mean:
A. Every day, you should imagine that you are forced to sell it; or
B. You should imagine that you will, every day, be forced to sell it.
(I teach law students to spot and fix ambiguities in draft contracts; this would be a nice practice example :-) )
OP imagines that there is a point in the future where they read the book without commiting to it. Selling/donating it would put an end to that possibility and force OP to imagine a future where they cannot read it, and hopefully helps them assessing one va the other, and prioritizing accordingly.
https://about.usps.com/notices/not121/not121_tech.htm
I've used it to buy/sell large tomes like this and it is really cheap ship things this way. It's not the speediest of delivery methods.
For example if you are interested in large integer arithmetic the 2nd edition of Volume 2 has some exercises that develop some methods that, if I recall correctly, I found more suited to my needs than the methods in the text when I was working on a large integer library.
The 3rd edition dropped those. I've long forgotten most of what I learned when working on my large integer library, but if I had to learn that subject again I'd probably actually start with the 2nd edition, and then check the 3rd edition to add to that.
I'd love to own them and be the kind of person to read them, but I know I would also struggle to give them the proper attention (not with the hundred other books to read on my shelf).
There is also a contrast between the two: they lie on opposite ends of a spectrum from shallow to deep, of things that make you really think / put in honest effort, and also give a deeper enjoyment.
As for reading other (good) books: like anything else, it's ultimately a question of what gives you most joy, I guess. Learning (say) real analysis from Rudin would also be deep and make you work, but Knuth has more jokes.
I don't get what you find strange about comparing reading TAOCP vs reading social media. Your time is limited, and each activity that you choose bears an opportunity cost. Of course it is an interesting question to ask whether you should read another book instead, but I bet that most people waste a lot more time on social media than reading a book that is not TAOCP, so svat's comparison does make a lot of sense in my opinion.
I read social media when I'm too tired to really engage with anything in a serious way.
I (try to) read TAOCP sometimes when I have a surplus of energy and ambition.
Admittedly, sometimes ambition comes along a little bit after I force myself to do something challenging, but TAOCP is so difficult for me that I really have to be feeling it to make any headway.
If you are interested in the underlying goal of Principia Mathematica, I urge you to check out the Metamath Proof Explorer (MPE): https://us.metamath.org/mpeuni/mmset.html
By itself, Metamath doesn't have built-in axioms. MPE uses Metamath to first state a small set of widely-accepted axioms, namely classical logic and ZFC set theory, and then proves a tremendous amount of things, building up to a lot of mathematics from careful formal proofs that rigorously prove every step.
Some things cannot be proven, but that doesn't mean that proof is dead.
Don’t get me wrong, PM is marvelous and there’s no gainsaying its enormous historical impact.
If your characterization of Metamath is correct, I don’t think that’s in the spirit of PM at all. One of the major problems PM had was the rejection of (what later became) the Axiom of Choice in favor of Russell’s convoluted theory of types. Indeed, that set theory (ZFC) is needed to get the rest of the way R&W were trying to go is one of the signal failures of the logicist program behind PM.
However, if you think the main point of Principia Mathematica was the very specific set of axioms that they chose, then that's different. The PM authors chose to use a "ramified" theory of types, which is complex. It does have sets, it's just not ZFC sets. Few like its complexities. Later on Quine found a simplification of their approach and explained it in "New Foundations for Mathematical Logic". There's a Metamath database for that "New Foundations" axiom set as well (it's not as popular, but it certainly exists): https://us.metamath.org/index.html
More Metamath databases are listed here, along with some other info: https://us.metamath.org/index.html
There are pros and cons for both systems.
The biggest problem NF has is, as usual, a social one: there just ain’t a lot of people working on, or interested in working on, NF compared to other set theories.
The elementary vernacular foundations of modern mathematics is (more or less) naive set theory; the starting tools of serious foundational work (as arcane as it is even within maths as a whole) are logic and more rigorous set theory, perhaps with some computability mixed in; the main tool of a mathematician who wants to reason in great generality is category theory (with some handwaving in the direction of the previous point about universe hierarchies and whatnot). There are some signs of convergence between these (and of mainstream mathematicians starting to once again take foundations seriously), but at the basic level those are what you’ll be dealing with.
None of them existed in their current form at the time PM was written, even logic (no Kripke semantics! no forcing! and no Gödel of course). Some did not yet exist at all. Some changed quite drastically in direct response to PM. And of course PM is the origin of (embryonic) type theory, which is the inspiration of the unified approach I referenced above. So as a historically important text, sure, if that’s what you want, but as a gateway to understanding more interesting maths it’d be terribly inefficient.
In that respect TAoCP was uniquely lucky. It was also a self-obsoleting book: it ceased to be comprehensive months after it was published, exactly because it told you everything there was to know about algorithms to date. Yet none of the stuff that’s in it is itself obsolete, there’s just immesurably more stuff now. PM, on the other hand, was attempt at “rationalization” in the 19th-century sense, and mostly a failed one except for serving as fertilizer for all of the later ones.
But then again, as a result of all this work from him, pulling together and categorizing and cataloguing and analyzing all the disparate ad-hoc “tricks” that programmers had come up with, Knuth basically put (this area of) computer science on good foundations and made it a respectable academic discipline. So in that sense he successfully executed a Principia-like project.
To put it another way, the computer programming/algorithms world is generally divided into hackers who derive satisfaction from elegant ways of bending computers to their will, and theorists who are happy to talk asymptotics, treat their problems abstractly as a branch of mathematics, and often never write a single program. What sets Knuth apart is that he straddles both words: is firmly a hacker at heart, but is able to bring prodigious theoretical skill to bear on the area.
---
PS- not to be "that guy" but I do think that a future edition should port the MMIX to RISC-V because it'll be actually runnable and I'd bet money that RISC-V is the educational standard going forward for the next 50 years.
In 1956, Knuth graduated high school and entered college, where he encountered a computer for the first time (the IBM 650, to which the series of books is dedicated). He took to programming like a fish to water, and by the time he finished college in 1960, he was a legendary programmer, single-handedly writing several compilers on par with or better than professionals (and making good money too). In 1962 when he was a graduate student (and also, on the side, a consultant to Burroughs Corporation), the publisher Addison Wesley approached him with a proposal to write a book about writing compilers (given his reputation), as these techniques were not well-known. He thought about it and decided that the scope ought to be broader: programming techniques were themselves not well-known, so he would write about everything: “the art of computer programming”.
This was a time when programming a computer meant writing in that computer's machine code (or in an assembly language for that machine) — and some of those computers were little more than simple calculators with branches and load/store instructions. The techniques he would have to explain were things like functions/subroutines (a reusable block of assembly code, with some calling conventions), data structures like lists and tries, how to do arithmetic (multiplying integers and floating-point numbers and polynomials), etc. He wrote up a 12-chapter outline (culminating in "compiler techniques" in the final chapter), and wrote a draft against it. When it was realized the draft was too long, the plan became to publish it in 7 volumes.
He had started the work with the idea that he would just be a “journalist” documenting the tricks and techniques of other programmers without any special angle of his own, but unavoidably he came up with his own angle (the analysis of algorithms) — he suggested to the publishers to rename the book to “the analysis of algorithms”, but they said it wouldn't sell so ACP (now abbreviated TAOCP) it remained.
He polished up and published the first three volumes in 1968, 1969, and 1973, and his work was so exhaustive and thorough that he basically created the (sub)field. For example, he won a Turing Award in 1974 (for writing a textbook, in his free time, separate from his research job!). He has been continually polishing these books (e.g. Vols 1 and 2 are in their third edition that came out in 1997, and already nearly the 50th different printing of each), offering rewards for errors and suggestions, and Volume 4A came out in 2011 and Volume 4B in 2023 (late 2022 actually).
Now: what is in these books? You can look at the chapter outlines here: https://en.wikipedia.org/w/index.php?title=The_Art_of_Comput... — the topics are low-level (he is interested in practical algorithms that one could conceivably want to write in machine code and actually run, to get answers) but covered in amazing detail. For example, you may think that there's nothing more to say about the idea of “sequential search” than “look through an array till you find the element”, but he has 10 pages of careful study of it, followed by 6 pages of exercises and solutions in small print. Then follow even more pages devoted to binary search. And so on.
(The new volumes on combinatorial algorithms are also like that: I thought I'd written lots of backtracking programs for programming contests and whatnot, and “knew” backtracking, but Knuth exhausted everything I knew in under a page, and followed it with dozens and dozens of pages.)
If you are a certain sort of person, you will enjoy this a lot. Every page is full of lots of clever and deep ideas: Knuth has basically taken the entire published literature in computer science on each topic he covers, digested it thoroughly, passed it through his personal interestingness filter, added some of his own ideas, and published it in carefully written pages of charming, playful, prose. It does require some mathematical maturity (say at the level of decent college student, or strong high school student) to read the mathematical sections, or you can skim through them and just get the ideas.
But you won't learn about, say, writing a React frontend, or a CRUD app, or how to work with Git, or API design for software-engineering in large teams, or any number of things relevant to computer programmers today.
Some ways you could answer for yourself whether it's worth the time and effort:
• Would you read it even if it wasn't called “The Art of Computer Programming”, but was called “The Analysis of Algorithms” or “Don Knuth's big book of super-deep study of some ideas in computer programming”?
• Take a look at some of the recent “pre-fascicles” online, and see if you enjoy them. (E.g. https://cs.stanford.edu/~knuth/fasc5b.ps.gz is the one about backtracking, and an early draft of part of Volume 4B. https://cs.stanford.edu/~knuth/fasc1a.ps.gz is “Bitwise tricks and techniques” — think “Hacker's Delight” — published as part of Volume 4A. Etc.)
• See what other people got out of the books, e.g. these posts: https://commandlinefanatic.com/cgi-bin/showarticle.cgi?artic... https://commandlinefanatic.com/cgi-bin/showarticle.cgi?artic... https://commandlinefanatic.com/cgi-bin/showarticle.cgi?artic... are by someone who read the first three volumes in 3 years. For a while I attended a reading group (some recordings at https://www.youtube.com/channel/UCHOHy9Rjl3MlEfZ2HI0AD3g but I doubt they'll be useful to anyone who didn't attend), and we read about 0.5–2 pages an hour on average IIRC. And so on.
I find reading these books (even if dipping into only a few pages here and there) a more rewarding use of time than social media or HN, for instance, and wish I could make more time for them. But everyone's tastes will differ.