Earlier this week there was a thread on HN about Taos, another very interesting OS from that time.
https://news.ycombinator.com/item?id=9806607
Today, with multicore, heterogeneous cores, a huge range of communication types, IoT and the physical world becoming connected from the tiniest of pieces up to the largest structures, there should be ample areas where OS research could find interesting challenges and solve problems.
The most interesting I've seen in a few years is the library OS work from MSR. But coming from Oberon, the most interesting feat was doing it to Windows and get it to work. Unfortunately it doesn't seem to become part of Windows.
To be fair though, the amount of virtualization tech developed the last decade could be perceived as OS related.
One professor I met once claimed that the Mach OS, micro kernels was the bane of OS research. Suddenly there was an OS easy enough to work on for a Phd to test implement a feature and get a degree.
The revivial in systems work has already started, and there is a lot going on eg see some of the talks in [1]. Projects like CHERI show you can do research on a real production OS (FreeBSD) not a toy one, while SeL4 shows you can even do correctness proofs on a small OS. Unikernels and other library OS projects and performance oriented projects are putting what was system code into userspace where it can be iterated faster, and moving to high level and scripting languages. Comtainers and microservices are causing a huge rethink of the monolithic architectures too.
And so on and so forth. Much stuff being done in directions that might actually achieve something. There's still hope for those of us wanting something other than monolithic garbage whose uptimes still can't beat a VMS cluster from the 80's and whose security is a measure of how convenient it is to hackers.
CheriBSD is more of a hardware research platform. We've had capability-based hardware and similar protection schemes for decades, but they never caught on.
Containers are an absolute disappointment the way they were popularized with Docker, and I think it's naive to assume that "microservices" are anything but a buzzword. The microservice architecture itself is exceptionally old and basically a reapplication of OO principles to high-level software components.
Unikernels and libOS are kind of interesting, though.
I agree on containers and microservices. IT rarely learns the good lessons of the past but often repeats its failures. The first good containers were Burrough's apps where they compiled source against a good API with checks at interface level (compile-time & runtime) with full reuse of code in memory to avoid duplication. Hardware-enforced, optional isolation & duplication if you wanted a brick wall. That has both an efficiency and security argument. This new shit might be an improvement on typical Windows or UNIX/Linux deployment but seems to be improving in wrong direction.
Best example is still them standardizing on complicated HTTP- and XML-based middleware instead of simpler formats (eg s-expressions) with a grammar I can auto-generate w/ validation checks on TCP/UDP-based middleware which I can also auto-generate w/ validation checks. Designing robust, efficient solutions with mainstream stuff is like walking through a friggin' minefield of problems.... where it can even be done! Starting to think they're all closet masochists...
I encouraged Microsoft Research to continue investing in tooling and verification tech that represents The Right Thing approach to systems. That pieces of it drift to their commercial products is even better. :)
The unikernels work going on.
Android's Java userland.
Apple's hybrid approach with containers.
This is one of the topics I actually agree with Rob Pike. UNIX is past its due date. There is so much to explore, specially since POSIX just perpetuates C insecurity model.
But, it all stopped. Research budgets dried up, the computer got bigger than any one OS team could handle, and some OS out of Finland came in and scooped up all the mindshare. Since then, we've been stumbling in the dark, using scant shards of the tools they made, like python, with overall systems not much better than what they have. Lispers talk of the AI Winter, but the ongoing OS Winter is far, far worse in terms of the absolute stagnation of an entire force of society. We really do deserve better than what we have now.