And if we are talking high end schools I've also read it's not simply a matter of getting admitted and that the rates of affirmative action candidates drops outs at much higher rates, which hurts/delays their chance to succeed in the future because they would have been better off dominating at a lower percentile school.
It's not the role of students to delay or defer their own education out of some sense of guilt.
Sounds like OP certainly deserved it to me. Instead we have these perverse incentives to go to "elite" schools purely because of the school's name (not because you're really getting that much of a better education than you would in a different top 100 university). These schools end up with thousands of applicants who definitely deserve to get in, but they need to cull the group of perfect candidates somehow because of practical considerations like faculty to student ratio. Whether they discriminate using race, or implement a lottery system, there's plenty of candidates that deserve the entry but don't get in because of practical limitations.
I certainly didn't get perfect scores on the SAT and I certainly wasn't valedictorian.
This mentality is very gross, and treats life like it's a fair game with clear winners and losers. Life is messy, and you can still do every single thing right and end up worse off. Likewise, you can, and many people do, fail "up" into extraordinary positions of power due to no merit of your own.