I'm suggesting that you are statistically likely to miss out on a significant monetary 'Ivy League premium' over your career, and that you will certainly miss out on a social 'Ivy League premium', which could affect your social status, mating opportunities (as silly as that sounds), and more.

Because it is a statistical observation, it does not necessarily have any bearing on your outcomes as an individual--though it seems likely that it does.