There are several problems with choosing by merit.

One, you have to define merit and whether it really is merit. If a rich kid scores higher because he got an expensive private tutor that taught him how to do better on the test really more meritorious than the smarter poor kid who scored lower because she didn't have said private tutor?

Second, you have historically marginalized groups that lose out because their ancestors were systematically abused, not allowed to be educated, not allowed into most institutions.

Saying that it should all just be a meritocracy now is like someone that gets a big headstart in a race and then once that headstart is pointed out, they start calling for fair play.

I don't support getting rid of standardized tests to undermine the rich because standardized testing REDUCED the wealthy's grip on higher education which before standardized testing was very much restricted to the wealthy.

Even in a racial quota based admission system, the elimination of standardized testing from each racial "tranche" only helps the wealthy. It's simply more expensive to fill a admission full of impressive extracurriculars and such than to do well on a test. Standardized testing was brought in to favor the intelligent over the wealthy to begin with, and while test scores and wealth are correlated this is in part because the entire point was to give better testers access to wealth and test taking ability is heritable.

So solve tutor and education access issues earlier in the education process. Everyone suffers when the world’s best institutions aren’t allowed to make the most of the world’s most promising graduates.