SPICE solved this (years ago in fact) by doing various sorts of compression, including detection of video, and streaming that over the network. (https://www.spice-space.org/features.html)