Did you ever see those 90s commercial Unix workstations? Like from SGI?
How about a Nokia N900? X on a smartphone. It ran pretty well. Today's smartphone hardware would run circles around that thing so I can only assume if somebody ported X to those it would also do fine.
I feel like people see things they don't like or imperfections and bash the whole thing as useless. X had a lot of mileage. I use it frequently today and don't see a lot of problems.
(1) X11 is not a window manager,
(2) There have been several really effective window managers for X11,
(3) MS-Windows has never been good at remote display.
There were no financial driver to make X11 a more optimal local solution.
There were huge financial incentives for MS to make their terminal-server offering more appealing.
And that’s all you need to explain the outcome.
Then X11 never managed to be a really effective window manager on a local workstation while Windows became really, really good at remote displays.