I think I understand the intuition behind that... in other words... coding an os kernel seems to have less entropy, less degrees-of-freedom, less subjectivity ... than frontend programming like painting GUI pixels and Javascript-framework-of-the-month.

But there's a lot of subjectivity and philosophical debates about low-level os design:

- the famous "Worse is Better" essay tries to explain difference between "New Jersey approach" (Bell Labs UNIX) vs MIT approach of error handling in a system routine [1]

- famous debate between Linus Torvalds and Andrew S. Tanenbaum about microkernels vs monolithic kernels [2]

- David Cutler (designer of VMS & Windows NT) criticizing UNIX i/o architecture[3]

- even lower level than o/s is cpu design where some criticize RISC-V not having arithmetic overflow traps in the minimum base specification. And in the 1990s, the RISC (MIPS philosophy) vs CISC (Intel philosophy) was a big debate.

[1] https://dreamsongs.com/RiseOfWorseIsBetter.html

[2] https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...

[3] https://retrocomputing.stackexchange.com/questions/14150/how...

I don't think that's the right takeaway. A decade ago I worked as an SAP consultant, and did some payroll projects in it. The display part of it was easy. All they wanted was a great big table display, the moral equivalent of a read-only Excel spreadsheet. No faffing about with CSS or Javascript or whatever.

No, the real complexity was the freaking business logic. Office workers, executives, contractors all had different rules for how their pay was calculated, there were all sorts of line items that applied to different classes of employee differently. The company had spent years building those reports by hand and there was a whole lot of organisational knowledge hidden in their heads, corner cases and exceptions all over the place. The sort of thing you can navigate in your head relatively easily if you've been doing it for years on end, but a newcomer has no chance of understanding - so a lot of iteration was involved in getting where we needed to be.

Put fiddly business logic and rapid iteration together, and you're burying yourself under a massive pile of spaghetti code in no time, flat. Trying to keep all of that sane both for myself and for whomever came after me was challenging in ways that none of the more systems-y work I've done since has ever replicated.

In my experience OS kernels just make more sense than most front-end code, regardless of the kernel design. There may be different plans for each kernel, but there is a plan. Major things change less frequently, changes are better thought-through, and people think more about what they're doing. Probably this makes economic sense, and the costs of kernel breakage are that much more than the costs of a broken UI, but definitely the kernel's design is easier to make -- because it has a design, rather than ever-shifting chaos.

Without it, UNIX would have been as esoteric as anything else and most likely would have died instead of sprung multiple clones.

But. There's another, much bigger difference between Unix and VAX/VMS/WinNT: The way I/O operations are handled in the kernel itself. In Unix systems, the kernel just passes I/O operations directly to the driver code, which immediately tries to do the operations on hardware. It's a very microcontroller-ish view of the world. WinNT treats I/O operations as sort of a "packet" which encodes the operation to be performed. That packet passes down through the layers of the kernel, including 3rd party modules and eventually drivers, until the operation is performed on hardware. Then, another packet percolates back up the stack to your app with the result.

Things like antivirus, dropbox, etc. can insert themselves into the stack and intercept/modify/reroute I/O operations and results globally.

I think that Linux probably has something like this, or you can probably cobble it together by using async I/O, io_uring, and/or eBPF.

Dig around in C land and you’ll find 100s of opinions, even more libs and toolkits that came and went out of fashion.

Web stacks are “computing” to industry right now. In the 80s and 90s it was OS and database.

If low-level had the old economic attention, we’d see the same activity and mess as we did in the 80s and 90s. Everyone has a compiler to tell you about, and a build system DSL that’s just right for your problem!

But big corp largely controls that and so a new layer of abstraction was added via web, and we’re seeing a shift now to less UI, and machine driven decision making that just gives us the next step in the recipe.

Frankly I don’t mind technology for navigating human society; buying stuff and literal navigation.

I still unplug with musical instruments and the forest.

The addictive applications we’ve built in technology; games, social and multimedia, got really banal for me a long time ago. If I hear things like “Itsa me, Mario” or something about Halo I recoil at the cringe.

I guess I don’t want my psychedelic noise wrapped up in branding and consumerism.

A logical error in an accounting package can probably often only be found by a checking tool that has the equivalent of all the functionality of the package under test.

OS bugs are more likely to have the side effect of violating language invariants (which eg KASan, UBSan and KCSan often detect), or violating protocols (so you can stress-test eg a TCP implementation and detect the bug), or it can violate API invariants (so you can use some API-level stress test like stress-ng to detect the bug.)

All of these tools are much less complex than the software under test. I think the warlord of Wu made the comment about debugging before massive automated testing, compiler instrumentation etc were a thing.

And I think a similar argument can be made wrt software and hardware-level visibility/tracing mechanisms - again an OS will end up being easier than higher-level software given a good investment in tooling.

The point is that when you compete with the best in the world in any area then it will be hard to win. Low level code is only easy when low level code isn't an important part of your product. For example backends to web apps, you don't write any low level code for them. The networking is handled via application server frameworks, the data storage is handled via pre packaged database programs so you don't even have to open a file etc. It isn't low level, it is just high level glue code between different libraries. This works just fine since you aren't trying to compete using low level code, so using off the shelf products without almost any code written by you is fine. But when your product requires you to compete using low level code then it is really hard.