IMO this is bit of a "the chicken or the egg came first?" problem.
The industry had a lot of needs in terms of computing when computers started becoming physically small. The very first few things -- OS, any tech paradigmae, chip designs, shell architecture and what-have-you -- that seemed to do the job good enough got 99.9% of investments and mindshare and thus even if everything we currently use is ugly and barely working, there's no economic interest to revisit the other competing standards from the dawn of computing and see if there are golden geese hidden in there somewhere.
So when a carpenter tells you "I need a natural wood and screws because that's what works best" you really should be reading that as: "nobody tried to invent better materials and joinery techniques or tooling so I'll stick to what I know".
"Good enough" is not a slogan to follow.
Sadly, I'm not aware of anybody working on such a project. Most people making "OS"s these days either just make yet-another-linux-distribution and confirm that old saying about the definition of insanity, or they're content to make some completely academic kernel that no one can use for anything but academic purposes. I wish I had the skills to start one myself.
The Lisp-all-the-way-down part doesn’t seem as useful as it was originally. You could get 80% of the functionality with 10% of the work of a full OS.
I think there is, perhaps, value in a middle ground. You could take the Linux kernel (chosen here for its existing driver support, openness, stable syscall ABI, and presence of virtualization and namespacing features), but throw away basically everything on top of it and build a new "operating environment". Then, over time, you could fork the kernel away, ditching legacy syscalls and features, maybe creating a stable driver ABI finally, etc.
You're missing the point. If the only goal was "functionality," a quick-and-dirty hack on top of existing systems would suffice.
However, the GP wants a system that's reasonably comprehensible all the way down, so he can have some confidence that it's not working against him in some way. The accidental complexity of decades of legacy software works strongly against that.
Hit the nail on the head.
Things are impossibly complex and convoluted nowadays. And working against us more often than not -- as you said.
It feels like we really need to start over. All the way down to the CPU and GPU hardware architecture.
Realism is often just a mask over defeatism.
Acknowledging the huge amount of work that needs to be done for an alternative to be adopted is not defeatism, it's realism. There have been plenty of minority, research, or experimental operating systems over the past few decades. Hardly any of them persisted, because it's a very Darwinian winner-take-all market.