But Universities are big - surely, you can find people who are of the same world view. Again, at Rutgers, we had ~40k undergrads. Given that many people, it wasn't all that hard to get together a suite of 6 math majors, hang up a huge whiteboard (technically, shower wall from Home Depot), and geek it up :)
As far as classes, it gets a lot better after you finish the common pre-requisites (if your school doesn't offer "honors" sections of those). When you have the choices of picking one of a dozen classes each taught by a couple of different professors (I'm thinking across-semesters), smart people often start gravitating toward the same teachers - ask around.
Hopefully this'll be the first semester where I'm really going interesting stuff. One of my first-semester professors let me and a friend into one of his top-level courses, so I'm excited for that.
When I'm in a beginner's coding class, sitting next to a girl who doesn't know how to program and hates the idea of programming, and she'd rather copy my stuff, then it lowers my ideal of what the class ought to be. When the professor cares more about making a joke and talking about her son than she is about teaching as effectively as possible, it lowers my ideal.
Not everybody's like that - I love my roommate and some of my floormates - but my point was that college is not for everybody. For instance, at this moment it's not for me. I've got an idea for something I want to do with my life - not my whole life, but my life right now - and I'm benefitting from planning parts of my life around that idea, and seeing it grow, and possibly seeing it succeed. After that, I'll probably end up back in college, but older and wiser and surrounded, hopefully, by people who've gone through similar experiences.
It may be different in later years, but that's my perspective as a freshman. College is built around the social. If it wasn't, then it wouldn't matter as much.