I think you've got a valid point. I would add that genetics plays a part. I grew up the same way, but my family didn't have much money. However my parents were former athletes and ensured my siblings and I took part in a regular physical activity (in one form or another) from a young age. I think that staves off some of the aggression -- if you don't look like they could just give you a knock around regardless of your demeanor.

That and I was often told I looked angry or sad while I spent my spare periods or lunches reading. Woops.

I think this at least applies to younger ages.

edit: wanted to add, I'm sure this changed some time ago. I'm referring to the 90's and early 2000's in particular.