That being said, there are several such "modular" kernels implemented on top of Xen. For example Mirage is a framework for creating free standing Ocaml applications on top of Xen.
You get all the NetBSD drivers, IP stack etc that you can use, just add memory management, setup and so on, which is not much work, and link your application in to it. It is still under development but the basic stuff is all working well.
Then there's bugs and debugging. If you have a bug in your disk driver you can slowly lose data without noticing. Quite a lot of other bugs will lock your system and you need to debug painfully from whatever logs you can find. Building in a non-C language might make this a bit easier but won't save you from the hardware locking up.
Once you've done this, what have you gained? You have an OS with no userland and no applications. What bits of Linux/BSD do you actually want to replace with something else? The code is fairly modular and OSKit exists to help with this.
The nearest to what you describe is probably retrocomputing; people building their own considerably simpler systems from the ground up. If you want a simpler platform to start on without having to first build the platform, consider Raspberry Pi; the bootloader there effectively just pulls an ARM ELF executable off the SD card and starts executing it. Have a look at https://github.com/dwelch67/raspberrypi
First have to decouple Go from its runtime before you could build a kernel in it, not to mention everything else that goes w/ writing a kernel. Similarly, you can't use significant parts of C++ when building a kernel w/ it.
It seems like a lot of people are looking at Rust as a system programming language, though... (google rust kernel)
Even though you can easily implement custom HTTP server you cannot easily implement full custom Go runtime.
If hard, why so difficult? I'm more referring to a theoretical reason, rather than practical.
Practical Reasons: - drivers are required - C/ASM is hard (ie lack of contributors) - different hardware architectures need to be supported. That part I get.
Why in theory can you not have an operating system framework that allows people to "relatively easily" build a custom kernel (instead of using Linux) the same way something like Go allows me to "relatively easily" build a custom HTTP server (instead of using Apache)?