But Universities are big - surely, you can find people who are of the same world view. Again, at Rutgers, we had ~40k undergrads. Given that many people, it wasn't all that hard to get together a suite of 6 math majors, hang up a huge whiteboard (technically, shower wall from Home Depot), and geek it up :)

As far as classes, it gets a lot better after you finish the common pre-requisites (if your school doesn't offer "honors" sections of those). When you have the choices of picking one of a dozen classes each taught by a couple of different professors (I'm thinking across-semesters), smart people often start gravitating toward the same teachers - ask around.