Yeah, no - X11 forwarding a video is at best "eh" when you're network local, and horrible otherwise. X11 forwarding burns a lot of bandwidth, and even when that bandwidth is available, going over TCP and SSH makes it implicitly quite latency sensitive.
It's not even fair to compare it to waypipe's h264 compressed buffer feeds.
Latency (how long it takes for messages to travel back and forth), which is where X11 over the network has major issue, doesn't matter in the specific case of watching a video.
Essentially, once you press 'play', there's just a torrent of data flowing down from the server to the client with very little need for two-way banter between the machines.