The part that is counterintuitive to most people when it comes to the "server" terminology is that, with X, your end-user workstation (which may be an incredibly dumb X terminal) is the "display server", which means you remote into a server (in the traditional sense) elsewhere, which then acts as an X client by making requests to your local machine to display windows.

The way most people think about it, "client" is your local machine and "server" is the remote machine that has lots of applications and is potentially multi-user, but X turns that backwards. The big iron is the client and the relatively dumb terminal is the server.