You can see then how it all neatly comes to fit. The environment you are placed in contains a certain type of people. Colleges help you connect with these people in this environment. It's just so "structured" that way. You become connected with mentors (your professors and alumni) and you connect with peers, and you connect with protegees. It's the whole neat, clean package.
No it is not impossible to meet and connect with people and build something great (as many have done) but college just streamlines the process.
You don't pay to go to college to learn. Just as you pointed out the self-motivated will ALWAYS find a way to learn. And to add to that, think about just how much abstract and intangible things that are SO IMPORTANT to a good life that you can't learn in college, you can't learn anywhere. You have to learn them through your own conscious efforts toward self-improvement.
The poster child for this is the idea of "academic" work as being less valuable (in dollars, anyway) than work done in a business environment. It's more than just cultural — colleges don't suffer under market stress in quite the same way. Part of that effect is an appreciable tuition, walled garden style.
So then you have to question what an academic "floating" environment can provide. If half the reason you're there is because you're already motivated to succeed without being there, what do you gain? Does Ivy League provide that in greater amounts? How much do connections and environment really matter?
I feel like that might be the stronger question that'll lead to informative answers.