It surely comes from television and movies, that's where I first got to know it. And initially I thought that this is just a media illusion.

But seeing real conversations using those stereotypes ("I was your typical nerd kid at school") here, on reddit and even talking to people IRL convinced me that this "caste system" does actually exist in real life (of course not to the overblown degree shown in movies).