Nothing. Any sufficiently complex system or body of knowledge is too large for any one person to understand. Does any physicist know ALL physics that was ever done? Does any one engineer know how to build a Ferrari? Let's be honest, we take for granted a lot of stuff that we don't fully understand, Linux is a major piece of infrastructure, it's natural nobody can fully wrap their head around it.
> Reading real programs is a part of learning computer science that should be emphasized more.
I had never thought of this in an educational sense, but I definitely agree. I sure wish I was taught more of that, toy problems get boring quickly.
Joe and Jane don't need to understand what you are doing, and you don't need to understand what they are doing. Of course partial understanding, especially for related technology is helpful, but the farther away you get the less you know, and the more you need to rely on others.
It might even span across to other niches. "Baking your own bread" is not that hard, if you get to it. But, becausethe entire process (fertilising ground, selecting seeds, growing grain, milling it) is too much for one person, lot's of people just give up alltogether and conclude that "today it is impossible to make your own food". Especially once you learn that bread mostly comes from large, complex industrial complexes today. Yet "just doing it" is very rewarding, as all the home-bakers will tell you.
And it spans across time as well: Sure, no-one can make a toaster from 0: mine own ore, distill your own plastics etc (there's a brilliant documentary from someone attempting just this, forgot its name). So people conclude that when the toaster stops working, there's nothing one can do, and they turn to Ali-express or Amazon and just buy a new one.
In short: I think labor alienation is toxic, in that it seeds itself into areas that don't need hyper specialisation.
The Linux Core Kernel Commentary [1]
It takes only the parts of the kernel you need to understand, leaving out the drivers, device classes, and much of te ancillary pieces. For the parts it chooses, it goes through them line by line and follows the boot process, explains the concepts of a Unix and SYSV-like kernel.
The 15M lines becomes much less intractible.
[1] https://books.google.com/books/about/Linux_Core_Kernel_Comme...
I suppose you could say many things have failed due to no one seeing the big picture among all the tiny details but without people focusing on the details none of that would have been possible to begin with.
I'm not a historian but I can imagine that a medieval specialist, such as for example blacksmith, could very much fix his own house (or maybe even build a new one), grow food, take care of animals, make wooden tools, make some simple herbal medicine etc. I'm guessing a lot of he was consuming was produced by himself and his family. Today, for engineers it's the total opposite - we only consume things produced by others, and essentially by strangers whom we've never met.
Regarding fragility specifically, if a major crisis comes (say a war), the desk jockeys, esp. ones didn't have the wisdom to accumulate resources, will fare much worse than someone who's say running a homestead. If they did accumulate resources (provided they don't become worthless by hiperinflation etc.), they will be able to buy the essentials and hope their stash lasts until the end of the crisis.
You should read Collapse Of Complex Societies by Professor Joseph Tainter.
Tl;dr a step change in complexity of ancient societies more often than not lead to their downfall. They were unable to train/retrain specialists quickly enough to deal with surprises.
>> The social impact, for one, is in general a large extent of estrangement from our human essence. Tangibility is intuitive, abstraction, not so much.
Specialization does introduce fragility, but that's something you brought to the discussion, not something that was present in the comment. This seems more like the common complaint that working on abstract fuzzy intangible things is damaging to your soul.
Using it as a counter argument against specialization isn’t very insightful though. Not even Marx tried to argue that (to Marx specialization was only alienating when you did it for the benefit of the bourgeoisie). Specialization is one of the most fundamental components of a developed economy. If you look at any ancient agrarian society, they had barely any specialization compared to a modern developed economy, and if you track the economic development of any society, you’ll notice specialization ticks upward along with it.
On the other hand, you make a fair point - tangible contribution to Things is probably built into us from our origin as tool-makers.
But isn't the gradual separation from direct contact with Made Things inevitable? At some point the body of knowledge required to make a meaningful contribution grows so large that learning it approaches the human lifespan. How can we avoid too much hyper-specialization in that case?
As a kid, it was an easier thing to reason about (wrongly). As an adult, I’ve concluded that it was never possible to both have something we’d think was civilization and for one person to know substantially everything humans knew.
The basic idea that Alan Kay has presented in his many similar YouTube presentations (most of which run with that office application) is what we call today domain-specific languages, you write the rules for antialiasing a pixel in whatever the ideal language would be for those rules: you just imagine that you have a dream language and "wishful think" the solution: then you go and you reimplement that DSL in another language that you are wishfully conjuring, a language-language, and that final one can become self-describing and can compile and optimize itself.
This is also a method of programming advocated in MIT's old SICP lectures, which will also cover the implementation of Lisp in itself, which is sort of a prerequisite for it.
That all said, I would love to read the source code someday but I can't seem to find it anywhere, the software product may be the KSWorld that was described in [1].
More careful reading of the proposal and reports will easily reveal this.
The 20,000 lines of code was a strawman, but we kept to it, and got quite a bit done (and what we did do is summarized in the reports, and especially the final report).
Basically, what didn't get finished was the "bottom-bottom" by the time the funding ran out.
I made a critical error in the first several years of the project because I was so impressed with the runable-graphical-math -- called Nile -- by Dan Amelang.
It was so good and ran so close to actual real-time speeds, that I wanted to give live demos of this system on a laptop (you can find these in quite a few talks I did on YouTube).
However, this committed Don Knuth's "prime sin" : "the root of all evil is premature optimization".
You can see from the original proposal that the project was about "runnable semantics of personal computing down to the metal in 20,000 lines of code" -- and since it was about semantics, the report mentioned that it might have to be run on a supercomputer to achieve the real-time needs.
This is akin to trying to find "the mathematical content" of a complex system with lots of requirements".
This is quite reasonable if you look at Software Engineering from the perspective of regular engineering, with a CAD<->SIM->FAB central process. We were proposing to do the CAD<->SIM part.
My error was to want more too early, and try to mix in some of the pragmatics while still doing the designs, i.e. to start looking at the FAB part, which should be down the line.
Another really fun sub-part of this project was Ian Piumarta's from scratch runnable TCP/IP in less than 200 lines of code (which included parsing the original RFC about TCP/IP.
It would be worth taking another shot at this goal, and to stick with the runnable semantics this time around.
And to note that if it wound up taking 100K lines of code instead of 20K, it would still be helpful. One of the reasons we picked 20K as a target was that -- at 50 lines to a page -- 20K lines is a 400 page book. This could likely be made readable ... whereas 5 such books would not be nearly as helpful for understanding.
Well anyway, thanks a lot for these presentations. I don't think it's as good as a software course where someone breaks all my preconceptions of what computing is, to leave me truly free, but they are helping to widen my thoughts.
Did some digging and found this site with some code, but I haven't looked closely enough to determine what it is specifically for... http://tinlizzie.org/dbjr/ And it is not exhaustive of what the PDF describes. Maybe someone is able to contact one of the original group working on this?
Here you go: http://v6.cuzuco.com/
I believe most of the Linux kernel source is actually drivers, or otherwise code for which you don't have the hardware to use on, so the actual relevant amount of code in the Linux kernel to e.g. a typical x86 PC might be an order of magnitude less.
I think you are going to like STEPS:
>The overall goal of STEPS is to make a working model of as much personal computing phenomena and user experience as possible in a very small number of lines of code (and using only our code). Our total lines of code target for the entire system -- from user down to the metal is 20,000, which we think will be a very useful model and substantiate one part of our thesis: that systems which use millions to hundreds of millions of lines of code to do comparable things are much larger than they need to be.
I can't agree. Linux is certainly huge, and yes it's size means it's beyond any newbie's capacity to understand it all.
But nonetheless lots of newbies contribute to it regularly. Most of them build up the understanding they need by reading the kernel code. Obviously, that means you don't have to understand all of it to get a lot of benefit from just reading a small subsection.
So it is indeed possible to go read the Linux kernel just as a learning exercise. The key point is you don't have to read all of it to get a benefit out of it. Reading just a small subset works just as well as reading the entire source of a small project.
https://jacquesmattheij.com/task.cc
3500 lines, micro kernels can be much smaller than that even.
But how much of it is the actual OS versus device drivers and other "extensions" of the main kernel? I assume there is somewhere a core that everything plugs into that has more or less the same functionality as what was done in the 9000 lines of the original Unix v6 code and only uses a fraction of all of the code of the kernel.
https://unix.stackexchange.com/questions/223746/why-is-the-l...
So it's still substantially heavier than would be easy to study, but you could probably still fit the core into your head (I've been able to deeply understand systems with similar LOC counts, though there's probably substantial differences in 'density' between codebases)
A debugger can also help. You can step through code as it runs. You can also often identify which code is associated with a feature, by putting breakpoints in likely places (dispatch functions!) and triggering the feature.
Wow, Unix is a work of art. I have hobby projects at 2x SLoC which are literally just small web services :(
File system: about 15Kloc
Network driver + IP stack: about 10 Kloc
Task manager, posix compatible libc odds and ends, maybe another 50Kloc, Graphics driver: 10Kloc, window manager 2500 lines.
So not that bad, you could do it by yourself in about 2 years of hard work, probably less than that if you use a VM instead of actual hardware if you're a halfway competent programmer. I've done it.
Perhaps we need something more modular, like a microkernel architecture :)
There are exactly zero reasons why this is the case other than a stupid ego war between Andrew Tanenbaum and Linus Torvalds who was gung-ho on recreating an OS from the 70's.
This irritates me as well, but blaming it on the kernel strikes me as absurd. Blame it on all the layers of abstraction and general lack of caring in applications instead.
It's true that there are cases where the Linux scheduler gets in the way of a good desktop experience, but desktop latency is too high even when the scheduler isn't a bottleneck. And if anything, good scheduling gets harder in a microkernel, not easier, because you have less insight into what is going on at a system level. (Ultimately, the distinction simply shouldn't matter though -- not for desktop experiences, that is.)
Yes, there are plenty of problems in user space. But those are not the root cause and without addressing the root cause you will not be able to solve the problem even if you don't have bloated software.
Lisps are hardly a modern invention. "Modern" languages do seem to be ever so gradually working their way towards full compile time metaprogramming though, gaining a great deal of expressiveness in the process.
Why can't distros be a package of microkernel + drivers?
The first week or so of the process was learning how to go from a new Debian machine (with the expectation we'd only used Windows or Macs before) through "git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git" (not on Github at the time) to finding the function called by "man 2 fork" and navigating the source tree/unmasking preprocessor stuff to show the actual implementation. Yes, you can easily make a wrong turn if you're trying to do that on your own. But the actual linux/kernel directory, which has most of the parts you'd want, isn't that much larger, and a lot of the difference is modern requirements like power saving and security.
Performance. Basically, the required amount of context switches (since all drivers run in user mode) impacts performance and cache use in a negative way compared to monolithic kernels. Whether this is still a significant issue with modern designs, I don't know, but that was the argument back in day (i.e. the early 90s when Linux came about).
They are built (usually) at the same time as the kernel, yes, due to the lack of ABI stability guarantees, but most drivers you don't use won't take up any RAM.
> Reading real programs is a part of learning computer science that should be emphasized more. As an undergraduate, we mostly programmed in BCPL, a local Cambridge language that is the fore-runner of C and C++. We had access to the full source code of the compiler and reading it was as valuable as the more theoretical aspects we learned in our compiler-writing course. Today, with open source, it is easy to read the source code for many systems actually in use (Linux, the Apache web server, Hadoop, TensorFlow, and thousands more) but these are hundreds of thousands, if not millions, of lines of code. As I said above, Linux is somewhere between 15M and 20M lines of code, depending on just what you include.
This is an interesting problem. Modern software is so huge and complicated that it's not feasible to go read it just as a learning exercise. Linux devs can probably spend years contributing to the project without having even visited all of its dark corners, much less understanding them.
Not sure what can be done. Though I am tempted to go see what a 9,000 line OS looks like.