Half the problem is that Linux's module interface is intentionally hobbled for non-GPL users, and in any case has no stable, versioned API. There'll probably never be something like a 'generic binary' Android, as there is Windows, that users just add some extra modules to, and that is by design.

Unless there is something I an missing, having a big kernel taking up RAM and disk space with support for hardware your phone doesn't have would be a waist of limited resources.

It would make porting CyanogenMod and other custom ROM's a lot easier to other phones, so I don't think that would be a waste. If manufacturers aren't going to support the devices anymore, at least the community will. Plus, it should make it a little easier for manufacturers, too.