Many people have done all sorts of stupid things, me included. For example, a few months back I was moping about a woman in London and not getting any work done, nor was I exercising.
Suppose someone comes along and asks why I'm not getting any exercise. The following is an incorrect reason: "...supply of time...probably working hard..."
The correct reason: I was sleeping until 2PM and hitting up the old monk before 6pm. I was banging a Rwandan pimpstress, a Ukrainian who got turned on by eve teasing and assorted other odd characters. Ganja played a role in this story as well.
None of this is work. It would be wrong to say I didn't exercise because I was working hard. Someone who says "chris wasn't working hard" does not have an axe to grind. They are simply correctly pointing out that enjoying ganja and unhappily married women [1] is not work.
[1] Tip: In India any woman over 26 should be presumed married. Learn from my mistakes.
It takes minimal imagination to extend these common experiences to a world where one is not momentarily hungry, or momentarily stressed, or momentarily taking shit from someone in power, but is rather subject to a constant onsalught of stressors due to poverty or systematic racism, and to imagine the consequences that such circumstances might produce. And yet you're not alone in your failure to do this minimal amount of mental work.
So as an aid to you in explaining the variance in the data, since your own life's experiences have seemingly left you with your current best explanation that a wide swathe of humanity is demonstrably and inherently inferior to the swathe of humanity that is currently winning, this book might serve as a useful meta-analysis covering the relevant issues:
http://us.macmillan.com/scarcity/
References inside.