It's not though. You have to fiddle around with ports and servers and environment variables.

But a drawing API is clearly not the way modern graphics works best. What would be really cool is a terminal that supported low latency streaming video. Then you can do proper GUI apps using any rendering system they want and it will properly integrate with ssh.