True, but that also goes for VNC (though VNC isn't as optimised as RDP is).

Modern RDP also has the option to turn (parts of) the screen into a simple h.264 stream so the connection doesn't die when you open up a web page with an animated background.

Seems like what we need is a specialized compression algorithm. Since the domain is largely constrained it sure seems like there's some low-hanging fruit here.

VNC with compression does already exist in the form of TightVNC[0].

[0]https://en.wikipedia.org/wiki/TightVNC#Encodings

Please stop already.

Lots of tradeoffs. If you invent a new codec, it's unlikely to make it into hardware for a while (even AV1 encoders are not yet as widely supported) and therefore you will have to do encoding and even decoding in CPU, which takes away resources from the workload. h.264 is still probably the best general purpose codec for real-time desktop streaming - low bandwidth requirements, 444 support, build to lossless, low latency, moderate CPU usage if GPU is not available, and well supported in GPUs for a long time (e.g. even back to Kepler).

You're right, but that's why SPICE has a so-called Agent that runs inside the guest and provides a neater integration (like seamless switching from client to host, ad-hoc USB passthrough, sharing the clipboard, detecting video streams (theoretically), ...) and that would be also an option for an RDP approach.