Athletic/gym gear, tank tops, bike shorts/leggings, speedos, etc. aren't really formal clothing, and they are practical both for physical activity and for fitness signaling (e.g. attracting mates and discouraging rivals.) (And sports from football to martial arts are a sort of ritualized fight.)

But I think you are on to something - men's attraction strategies are often via resources rather than health/fitness (or overt physical conflict with rivals.)