I think a lot of the things we were seeing at the OS level in the early 90s are now feasible to do in user space. That, plus the fact that it is prohibitively hard to talk to devices [1], has reduced the innovation in this area but also moved it elsewhere.

Having dealt with the pain of writing OS code, or libraries that have to function in one or more OS's, I can see why people prefer to innovate elsewhere if they possibly can.

That being said, there's a lot of possible "systems" work or work that could be described as "operating systems" in the broad sense that isn't in-kernel, and that work seems to have tailed off too.

[1] My favorite data point on this, which isn't even one of the more complex devices: back when I TA'ed CMUs "build a preemptive kernel" 15-410 class, the students wrote a kernel that would really boot. We had an old machine for their kernels if they wanted to see things working on a real machine not SIMICS. However, it had to be an old machine with a PS/2 keyboard, as the complexity of talking to a USB keyboard (or so I was told; I never verified this) was roughly equivalent to the "build a entire preemptive multitasking kernel" - and obviously far less pedagogically useful.