There's a lot of literature that point towards more application-specific optimization in the lowest levels of the operating system, for example the Exokernel papers are an early example of this. More recently, you may want to have a look at the the MirageOS paper(s).