They are for all practical purposes fulfilling the same role.
Additionally a Linux kernel that exists for the sole purpose to keep KVM running, while everything that powers a cloud workload are Kubernetes pods, it is nothing more than a very fat microkernel, in terms of usefulness.
The point is how Linux is being tamed to provide some of the concepts, in spite of its monolithic design.
But naturally we can discuss minutiae instead.
Presenting capabilities / namespaces to userland is a completely different and in the case of Linux, orthogonal thing to presenting capabilities/namespaces to kernel services. I guess you could argue that the concept of capabilities came from microkernels, but when it’s applied to only user space, it’s just not really related to a microkernel anymore at all.
That’s basically the whole problem with capabilities and especially their application in namespaces from a security standpoint in Linux: they try to firewall these little boxes from each other but the kernel they’re all talking to is still one big blob. And this difference is meaningful in a security sense, not just some theory hand waving. https://www.crowdstrike.com/en-us/blog/cve-2022-0185-kuberne... is just one good example, but entire classes of mitigations are rendered meaningless by the ability to unshare into a box that lets an attacker touch exploitable kernel surface area which is not further isolated.
Linux security being a shit show and undermining these efforts is kinda besides the point: they are still attempts provide a runtime closer to what microkernels would naturally provide in a backwards compatible way. Indeed, these containers could be turned into fully fleged VMs if there were the resources to make it happen.
To me the word "microkernel" expresses how the kernel is structured, not what userspace interface it presents. A microkernel is built by separating kernel services into discrete processes which communicate using a defined IPC mechanism. Ideally, a microkernel offers memory boundary guarantees for each service, either by using hardware memory protection/MMU and running each service as a true "process" with its own address space, or by proving the memory safety of each kernel service using some form of ahead-of-time guarantee.
Of course, doing this lends itself to also segmenting user-space processes by offering a unique set of kernel service processes for each user-space segment (jail, namespace, etc.), but there's no reason this needs to be the case, and it's by and large orthogonal.
I do agree with what I eventually understand the grandparent poster was trying to express, which is that running a bunch of KVMs looks like a microkernel. Because then, you've moved the kernel services into a protected boundary and made them communicate across a common interface (hypercalls).
But that's not how Kubernetes works by default and in the case of containers and namespaces, I think this is entirely false and a dangerous thing to believe from a security standpoint.
> They are ways of segmenting userspace in a more finely grained manner and they do make attacks harder.
From a _kernel_ security standpoint (because we are talking about micro_kernels_ here), I actually think namespaces make attacks much easier and the surface area much greater. Which is basically the entire point I was trying to make: rather than exposing fragile kernel interfaces to exclusively system services with CAP_SYS_ADMIN, you now have provided an ability (unshare) for less-trusted runtimes to touch parts of the host kernel (firewall, filesystem drivers, etc.) which they would normally not have access to, and you have to go back and use fiddly rules engines (seccomp, apparmor, selinux) to fix the problem you created with namespaces.
To be clear, I think from a big picture standpoint, it's a tradeoff, and I'm nowhere near as anti-container/anti-namespace as it may seem. I just get annoyed when I see people express namespaces as a kernel security boundary when they are basically the exact opposite: they are a kernel security un-boundary, and Linux's monolithic nature makes this a problem.
Nothing else, critical set of OS services are no longer hosted in a single process space, rather required a distributed system of processes running on a single computer node.
The moment these processes are able to be hosted in a set of computing nodes, we enter the realm of distributed OSes, is is another discussion.
The amount of services that remain on the mikrokernel, versus what is hosted in each OS process, running outside of the kernel, depends on each microkernel, there is no rule for this division of labour and each one that was ever designed has chosen a different approach.
If you cannot see the parallel between this and a swarm of containers running on a node, doing the actual workload and only requiring the kernel services due to the way KVM is implemented, and rather focus on a security mechanism that is detail on how Linux works, well, it isn't on me to further break it down.
That's not how Kubernetes works by default, so I thought we were talking about container runtimes. I still disagree strongly from both a practical security and theory standpoint if we are talking about container runtimes implemented using namespaces and cgroups.
[1] https://www.zdnet.com/article/minix-intels-hidden-in-chip-op...
“I remember running the QNX demo disc: an entire graphical operating system on a single 1.44MB floppy! Whatever happened to them?”
“They got bought by blackberry, which ended as you’d expect. QNX had a lot of success in automotive though.”
“Nowadays Linux and Android are dominant in new cars, though, proving once and for all that worse is better.”
exeunt. End Scene
Various sources state that they rebased to OSF Mach Kernel 7.3, which was based on Mach 3 and parts of Mach 4. The OSF MK ancestry of macOS XNU can still be seen in paths:
https://github.com/apple-oss-distributions/xnu/tree/main/osf...
The Darwin 0.1 and 0.3 releases contain the old kernel, derived directly from NeXTSTEP, and that's the direct derivative of Mach 2.5. The later XNU appears to be a reconstruction of that kernel with unencumbered code and that's also when IOKit replaced DriverKit.
EDIT: Apple calls out Mach 3.0.
https://developer.apple.com/library/archive/documentation/Da...
https://developer.apple.com/library/archive/documentation/Da...
Apple took Mach 3.0 and collapsed it into a single address space for performance purposes. Certainly reasonable.
True microkernels are, alas, more of an ideology than a practical reality, as the long-suffering GNU/HURD team discovered; Tanenbaum has been clear that the MINIX/NetBSD experiment was more about principles than performance. That said, certainly many hypervisors have attained success with configurations that coincidentally happen to be the same footprint as a microkernel.
[0] https://vita.militaryembedded.com/5159-open-virtualization-a...
HURD is Mach-based; it is stuck in a pre-L4 world and thus irrelevant to this conversation.
Very 40 years ago technology. Still better than ~60yr ago UNIX, although they're trying to ultimately be UNIX-like.
It is sad Mach and HURD still keep coming up in microkernel discussion.
The story goes that the early days were dominated by some ridiculously bad project management that makes Cathedral and the Bazaar era "Cathedral" a nice and well maintained project.
Most importantly, GNU Hurd lacked the "ecological niche" to drive work towards it because it couldn't deliver enough, just like lack of information about 386BSD led to Linux (Linus explicitly said that if he had known about 386BSD, he would have worked on it and maybe forked it)
NT and XNU never had microkernel origins - Cutler explicitly refuted this at a conference in the early 1990s
The alternative is a microkernel-based system, in which most of the OS runs as separate processes, mostly outside the kernel. They communicate by message passing. The kernel's job is to handle the message passing, interrupt handling, low-level process management, and possibly the I/O. Examples of this design are the RC4000, Amoeba, Chorus, Mach, and the not-yet-released Windows/NT.
That is why I thought Windows NT had originally been intended to be a microkernel.
Wikipedia has a decent depiction of the hybrid structure, but quoting from the NT Design Workbook:
> This specification describes the kernel layer of the NT OS/2 operating system. The kernel is responsible for thread dispatching, multiprocessor synchronization, hardware exception handling, and the implementation of low-level machine dependent functions.
> The kernel is used by the executive layer of the system to synchronize its activities and to implement the higher levels of abstraction that are exported in user-level API's.
Not only did the graphics stack moved again back into userspace, there is now a complete userspace drivers stack, and VBS (Virtualization-based security) for several kernel components that run on their mini Hyper-V island, talking via IPC with the rest of the kernel.
Likewise on XNU land, Apple has started a crusade already a few years ago, to move all kexts into userspace, no more drivers in the kernel beyond the ones Apple considers critical.
In both cases, they never were a pure monolithic kernel, due to the way walls were introduced with kernel level IPC to talk across modules, instead of straight function calls.
ProvenCore (https://provenrun.com/provencore/) has a proof that covers correctness (memory safety, and more generally absence of UB, termination, etc.), functional properties (e.g. the fork() system call is a refinement of an abstract "clone" operation of abstract machines), and security properties (memory isolation).
Edit: That being said, we should be building microkernel-based OSes, and if for some cases performance really is a restricting factor, they will be exceptions. The security, robustness, flexibility, etc. of microkernels is not to be understated.
What would you be able to do in a userspace scheduler that couldn't be done safely in an in-kernel scheduler? Why couldn't it be configured or given a safe API to interact with userspace at runtime? But I guess your shared memory bit is an API to a userspace scheduler?
> I understand the minimality principle but one should not be afraid to violate layers of abstraction when justified.
Rather, in a well-designed system, the justification necessarily is performance. Abstractions exist for the ease of the developer to write code that is correct. Performance (runtime, memory usage, etc.) is the concrete impact of code. For concerns like debugging or flexibility, those arise if the abstractions are lacking. Performance concerns arise because even good abstractions impose overhead. Note that conventional layered architectures are usually poor abstractions for implementation. Good for design but not coding.
> But I guess your shared memory bit is an API to a userspace scheduler?
Indeed. I would rather not, but shared memory probably is necessary to some degree for a userspace scheduler.
It is moving the higher-level things into userland that is the harder problem, and the one that has been challenging for microkernels to do well.
More to the point, drivers in userspace is an important distinction between "pure" monolithic kernels and microkernels: the former is optimizing for performance and the latter is optimizing for robustness. It's not about ease of implementation for either. It's quite meaningful to shift on the axis nowadays: it represents a critical pragmatic decision (notice purity is irrelevant). You're right that "higher-level things" such as the networking stack or filesystem are also crucial to the discussion. I think here, too, ease of implementation is not relevant, though.
The fact that this can be done with OS code not explicitly designed for this signals to me that there are no roadblocks to having a high-performing general purpose microkernel running our computers.
"Safe enough" measured by the standards of this upside-down industry...I'll let everyone decide that for themselves.
"More efficient": while monolithic kernels have a higher ceiling, currently there's plenty of OS research papers and industry work demonstrating that more integration with userspace brings performance benefits, such as in the scheduler, storage, or memory management. Microkernels encourage/enforce userspace integration.
"More scalable": I think this has less to do with kernelspace scale and more with how nodes are connected. See Barrelfish[0] for eschewing shared memory in favor of message passing entirely, running separate kernels on each core. Meanwhile Linux has gradually discovered a "big lock" approach is not scalable and reduced coarse-grained locks, added RCU, etc.. So I think we will go moreso towards Barrelfish. But on a single core, for what goes into the kernel, that's covered by everything but scalability.
[0] https://people.inf.ethz.ch/troscoe/pubs/sosp09-barrelfish.pd...
At present there's 16x64-bits of register state saved (128B), but if we were to have full support for the vectors, you need to potentially add 32x512-bits to the state (plus another 16 GP registers when APX arrives). Total state that needs moving from registers to memory/cache would jump to 2304B - a 1800% increase.
Give that memory is still the bottleneck, and cache is of limited size, a full context switch is going to have a big hit - and the main issue with microkernels is that a service you want to communicate with lives in another thread and address space. You have: app->kernel->service->kernel->app for a round-trip. If both app and service use the AVX-512 registers then you're going to have to save/restore more than half a page of CPU state up to 4 times, compared with up to 2 on the monolithic kernel which just does app->kernel->app.
The amount of CPU state only seems to be growing, and microkernels pay twice the cost.
In comparison, the average context switch time of Linux is already on the order of ~1,000 nanoseconds [1]. We can also see additional confirmation of the store overhead I computed as that page measures ~3 us for a 64 KB memcpy (load + store) which would be ~187 nanoseconds for 4 KB (load + store) where as the context saving operation is a 2 KB register -> store and a 2 KB load -> register.
So, your system call overhead only increases by 10 percentage points. Assuming you have a reasonably designed system that does not kill itself with system call overhead, spending the majority of the time actually handling the service request, then it constitutes a miniscule performance cost. For example, if you spend 90% of the time executing the service code, with only 10% in the actual overhead, then you only incur a 1% performance hit.
[1] https://eli.thegreenplace.net/2018/measuring-context-switchi...
While seL4 runs on x86, it is not recommended.
Effort seems to be focused on ARM and RISC-V, particularly the latter.
An argument could be made that x86's baggage makes it a particularly bad fit for offering the sort of guarantees seL4 does.