What is the real downside of that vs replacing android with linux? I could see there being more pro's than cons except for specific use cases.

Android kills stuff when it feels like it wants more room to cache apps in ram. The background stuff like the media indexer takes CPU time and generates heat (these are pretty easy to overheat when used like this.) I think some of the older android userspace even has flaws exploitable over the network. etc.

RAM and CPU usage are higher, because you're essentially running two OS's at the same time.