IIRC, the binfmt elf code in the Linux kernel basically only does enough elf parsing to map everything into memory and kick off the linker. Then the linker does the rest, in userland. It matters because the binfmt code can be very hardened and minimal, whereas the userspace can go and do whatever it wants.

Of course the ELF parsing doesn't literally have to be done by the kernel, but there's no obvious good place to put it otherwise, considering it's a critical bit. If security is the real concern, then why wouldn't you just drop privileges while parsing ELF instead? I don't see the benefit to moving this code outside of the kernel. Maybe somehow moving it to a different protection ring would be helpful; but same for the rest of the binfmts.

I believe the Windows loader that loads PE files, is in fact in the kernel, along with a lot of other stuff. At the end of the day, when the Windows NT kernel was architected, speed was really important and security practices were not matured. I mean, the kernel handles or at once handled a lot of stuff, especially related to the GUI and even network RPCs and network services.