Linux doesn't seem to be the bottleneck for neither performance nor stability in a lot of contemporary systems, though, particularly not if we're comparing against a new, hypothetical operating system.
Consider unikernels on VMs with paravirtualized hardware and thin hypervisors. When I boot my servers, hardware that's never used gets initialized. Filesystem code for filesystems that are never mounted is loaded. A whole lot of code that's specific to the CPU and chipset is patched in important execution flows. A unikernel has nothing but the code I want to run.
The fact we got it right most of the time, for most of the hardware combinations we use, does not make it any simpler, faster, or more reliable.
When I mentioned mainframe OSs, I was not talking about hypothetical new OSs but technology that has consistently excelled for decades now.
As pointed out before, mainframes have OSs that are exceedingly simple from the program's point of view and that offer a limited set of functions in exchange for extreme performance and stability.