The only way something like this would work would be the OS treating the heterogeneous compute resources as "first-class citizens" and part of the normal OS scheduling and resource management. I don't think any production OS today does that.

And just after I write it, an article on AMD's work on ROCm pops up.

Exciting times.