Why would it have to replace the existing drawing model? I'm thinking of a way for an app to ask the server if it can render a particular set of high level primitives, if the server responds that it can, the apps sends those instead.

That would allow existing apps and existing display servers, new apps and older display server, and old apps and newer display servers to work fine.

If both the app (at the toolkit level) and the display server are recent, however, remoting is massively sped up.