There are plenty of people of different social classes than you who don't work at the same company as you, so I don't think a social norm saying that you shouldn't ask out maybe a few dozen people you work with is a "closed caste system".