I second this explanation. Additionally, from a UX perspective, if a user wanted to join your server you had to give them IP/port/pw and they'd type it in manually. So even the users were accustomed to addressing servers. "Server" entered the community lexicon as "the place we chat".