Hard work only matters when you work hard to change yourself. If you work 80 hours a week bagging groceries nobody can say you don't work hard, but it isn't very productive work.

Most people just follow the path set for them, and among those who do set out to change they mostly just change from one standard path to another. Like switching from bagging groceries to college to become a programmer. Very few want to do the hard work required to change the world, they prefer safety, even among richer people.