Sure, minimal amount needed for safety should be in the kernel. That can also be somewhat federated if you like. Much of a kernel is not hardware access though, it is things like filesystems and sockets way on top of hardware.