Imagine college A that 100 apply for and all 100 get in vs. college B where 10,000 people applied and only 100 got in. If both colleges are choosing students based off of some measurable criteria, let's say the SAT score. Would you expect to have higher average scores at college A or college B? Obviously college B would have students with better SAT scores simply because they are so much more selective.

Assuming that the college is selecting students for attributes that you are looking for in fellow students, then the college that denies more people will have students with more of the attributes you are wanting to be around. So it isn't just about perception. There is some math behind it as well.