As someone who has actually tried writing a kernel in Swift, the issue is purely the runtime. While you can technically build a module without it needing to link to external Swift standard library binaries, the second you try and do anything with an array or optionals, you suddenly need to link in a 15MB behemoth that requires SIMD to even compile (at least on x64 and arm64). Porting this to bare metal is possible (and some have done it for a few microcontrollers) but its a pain in the ass.

I do love Swift and would use it for systems stuff in a heartbeat if I could, but there are also some downsides that make it pretty awkward for systems. The performance isn't always the best but it's (generally) very clear and ergonomic.