It used to be possible (and done) to have a thin client X server that would have the whole desktop running elsewhere. You could run a computer lab with a bunch of thin clients and a handful of desktop servers for lack of a better word. That was probably hundreds, maybe thousands.
Since then, though, X moved a lot more work from the Xserver to the clients; text is almost exclusively rendered on the client and passed as an image to the server to display.
If you did a classical experience, it might work ok to millions, but the user experience would have to be pretty meager. There's also the issue that there's not really a good way to sandbox different X clients; so nobody in their right mind would let a random internet server connect to their Xserver. Connectivity issues abound as well.