I think you've just got perceptual tuning for the human to human context. It's like how human faces that are 5% different can be very obvious to us, but it's much harder to tell (for example) two cats apart at the same difference level. All of your adaptive pressures are pulling in that direction.

But at a system-scale view of intelligence, humans are squashed together and it plays a pretty small role on outcomes. You should much prefer being rich than smart if your goal is 'success' by most metrics.