The only thing you've mentioned which is a barrier to entry (to elite schools, e.g. Yale) is legacy admission and donations.

The remainder are merely possible causes for non-elite people having lower merit. They are not barriers for a meritorious non-elite person to attend Yale or join the elite.