I also went to some very good schools and did not live in a rich area. It was a good area sure, you weren't going to hear gunshots on your walk to school, but it was not the "rich people" area. The rich people area actually had worse schools academically speaking. This makes sense when you think about it. We tended to have a lot of students whose parents were highly educated immigrants, whereas the rich part of town generally tended to be old money and not as focused on being hyper competitive academically.