In my experience, it depends. X can be a pain, but getting a compatible font and terminal behaviour on the client can be a pain too.
I'm not praising X nor am I shitting on TUIs, also I have indeed had more problems with X in general, but not when both host and guest were Linux machines.
So when someone makes a program that just decides to print "Hello" by just printing out those chars on my computer, with my font (or the default, the system, the fallback) it is kind of nice that I at least can see what it tries to tell me.
I'm not arguing that X11 is more streamlined than remote SSH (or tmux!) in a terminal / console. But it is far more available than you might think.
You probably also refer to Windows and MacOS, but that just corresponds to another type of difficulty (Xming, Xquartz), it doesn't make it inherently worse/impossible.