There's no difference between you and the wealthy folks in class, your brain works the same as theirs.
Socially, it's a whole other story. I could never afford to go out, and all my evenings were spent working at either a cafeteria or as a lab assistant, anything to pay the minimum payments on my loans. My classmates would invite me out with them for dinners and trips, but I never went, and eventually they stopped asking. My social circle became other students in similar situations, but it definitely made me feel like I didn't belong.
This social stratification was also present in class, when people had to form up for groups for group projects, since those groups tended to form along social group boundaries.
I don't regret going one bit, though. I have a great degree, paid off my debt, and have been working in preeminent silicon valley companies for more than 20 years now.