But the major toolkits have taken to rendering everything outside of X and just dumping the result as a bitmap into a box, that is absurd for all but the most local of work. And yet they complain about X being the problem.
X is not the problem. The problem is a generation of devs coming over from MS and Apple platforms and not recognizing that X is different. They just want to dump their eyecandy into the buffer and rely on Moore's Law for things drawing more than 1 pixel an hour.
Meanwhile, Linux still had blitted windows that left a grey trace for a brief instant when you moved them, and pixelwise window edges, and terrible transparency.
Came along Compiz, which reversed the way clients interacted with X, and which did the slick animations and fancy graphics of the mainstream OSes. What was the only way it found to do that? Rendering everything outside of X. And nobody could do it with the old design. So that design was quickly adopted by KWin and so on.
It turns out that nowadays screens are not lightweight clients over a serial line, and they have more than double the number of pixels of VGA.
Compositors became a necessity for the Linux Desktop to compete, reducing X to an IPC system that it cannot be good at, because it is not the situation it was built for. Wayland is built for the status quo.
Just about everyone who has worked seriously on Linux graphics agrees that X is a problem and Wayland is the solution. You might want to look into why they all say the same thing.
There's no conceivable reason (these days) that you'd want to send a "draw a stippled line from point a to point b" command to an X server or graphics card.