There's nothing inherently insecure about the X protocol. Sandboxing X would involve simply not providing information about drawables not created by the client. That means filtering out input events destined for those drawables and returning blank pixmaps when their contents are asked for.

That the X server doesn't do this, and provide knobs to control who is trusted and who isn't, is a shame. But again, we don't have to burn the entire graphics stack because security.