A smaller surface area with a stable HAL would separate the proprietary / often bad board support code from the kernel itself and allow core changes to progress without requiring software-adverse hardware manufacturers to keep up.
Whether or not this is a good thing is certainly debatable, but I'm not convinced the Linux "get to mainline or get wrecked" approach has stood the test of time well.
"Your WiFi only supports Android Kernel API 3.9. You need 7.0 to run this version of Google play".
The Linux kernel already has a stable user space ABI. If your hypothesis was valid, you would be able to upgrade the Android userspace on an old kernel with no problem. But you can't -- and that's with just one well specified ABI with a standardized, portable subset to worry about.