No, the big problem with X isn't the transport but that once a client has access to the X server it can do more or less anything it wants (including logging keyboard input and recording other windows).

And I haven't actually used it[0] but I thought waypipe was the better option for remoting?

[0] Well, successfully or recently; I'm assuming it's usable now.