I agree with this sentiment entirely. I went to a public magnet high school with SAT scores averaging around 1480 and at least half our graduating class every year ends up in an ivy / mit / stanford. I'd say more than half of those who made it to the top tier got in from legacy / extracurriculars / obsessive dedication to good grades and making teachers happy as opposed to genuine intelligence and love of learning. Also curiously, like you said, everybody from my graduating class who went to caltech, although not necessarily teh uberg3nius math wiz, was at least minimally competent in some technical field.

Although one could make the case that a lot of those schools, especially the ivies, are picking people based on their ability to lead and 'make a difference' in some general sense rather than pure raw ability. Therefore they may intentionally focus on apparently manifested soft qualifications like achievement in extracurriculars, personal discipline as exhibited by a flawless GPA, or well-established personal connections.

This is all strictly re: undergrads of course. I've never seen a single retard get admitted to a top-tier PhD program in science or engineering.