There are thousands of kernel variants between every Linux distro. You always need to recompile drivers. nVidida and AMD/ATI get by this by providing binary blobs and a shim layer than can compile and link the two.
In the Windows/Mac world there is one version of the kernel officially maintained and distributed by their respective companies. They can keep the kernel consistent for a given release. That's what you have one set of drivers for Windows 7, one for 8 and one for 10, instead of a different set of drivers for each and every service pack.
Android could, at a minimum, standardize the kernel for every Android version. With that, binary drivers could work across devices.
How do you want to boot your kernel? There is no BIOS or UEFI or other standardized firmware, that would initialize at least one CPU, RAM, storage and load your kernel from there. Every ARM implementation boots in different way.
After you boot up, how do you enumerate devices that your kernel can use? There is no PCIE or other plug and play bus, that can enumerate attached devices. Your OS has to know what is available. Poke a wrong port and your board freezes. This is the reason, why Device Tree is a thing.
Linux kernel does not require all drivers baked in - kernel modules existed for decades already. Monolithic does not mean that everything is one big blob. NT is also monolithic, for example.