Right, it makes a lot more sense to judge people on their adherence to a particular muscular profile than on their wealth.

Fitness is an indicator of physical health and ability to defend from predators and fancy cars and clothes are indicators of mental/social prowess and ability to garner resources. They're both valid, if imperfect, criteria by which to judge someone for reproductive fitness.