I had older classmates. I can't remember ever thinking of them as "losers." They were usually more interesting than the younger set.

No, you won't end up in a fraternity. However, there are a surprising number of people who end up in school years later. This shouldn't be a concern for returning.