You can answer that question by playing semantics - copying from a dictionary, 'prestige' is having 'standing or estimation in the eyes of people'.

Most of the prestigious people I can name aren't prestigious because of their academic achievement, that is for sure - so it isn't obvious why an academic institution should be prestigious full stop. They all teach a similar syllabus, and most professors are remarkably talented. Harvard is prestigious because the students that go to Harvard are well connected, not because Harvard is imparting some superior secret sauce to its students. The selling point is the network of other people who are important, eg, the children of the sort of people who went to Harvard.

We don't give people standing or estimation from academic or athletic ability except in extreme edge cases (much more extreme than college admission). There is a strong correlation between intelligence and prestige, but that is because smart people make decisions that earn them prestige more than because academic ability is prestigious.