I agree, and because that's a rather obvious market inefficiency, I see it as an opportunity for a startup to make money while improving the world.
YC may be closest to being this startup.
If their process was an "Empirical, blinded, skills-assessment", and they funded anyone who met their predefined requirements, we'd have a real alternative to class-based universities. At least for anyone willing to create their own business.
YC is already more fair (less gameable) than the elite universities, but it's still far from being fair enough or big enough to really change things.
Graduation rates at universities for the elite are nearly 100%. Getting into the club is the only hard part and everyone involved knows that.
Empirically, what's required to get into the club is being wealthy enough to afford the private schooling, tutoring, and extra circular activities necessary to help your child game the admissions process.
As long as people value these prestigious brands, the elite will continue to buy them for their children. It's got nothing to do with education itself, which could be obtained any number of ways already.