Exactly. You hit the nail on the head. It's not surprising at all.
The converse (inverse? I always mix them up) is true too. What would the jocks do if a nerd tried to come out for a sport? Generally make fun of, pick on, and possibly try to hurt them. This mentality exists within all groups.
I'm sure there's something really interesting to discuss here about sociology and the human psyche, but unfortunately I'm not well versed in those subjects.
And before anyone says, "Not everyone treated you that way," they did. The ones that weren't actively tormenting us were passively allowing it to happen while they were allowed into the groups. Everyone that wasn't against us was with us.
So yeah. We're a pretty closed group now. It shouldn't come as a surprise at all.