These people make up an infinitesimally small proportion of the general population, and as such don't make for a good counter-example.
More comfortable, I dare say, than the casual attire I must wear when working for west coast tech companies, lest I not conform to west coast tech company expectations. T-shirts and hoodies lose their charm when they start to feel like a requirement. Nice silk suit trousers feel nicer than blue-jeans, if they fit well they feel as nice as pajamas, but the looks you get in a west coast tech offices when you wear such clothing are withering.
In a less ideal world: As long as management doesn't get the idea to mandate dress clothes or GOD FORBID business casual, I'm fine.
Fuck business casual. I will never ever ever ever wear a polo shirt. Polo shirts are for old hairy guys that play golf.
However, I think there’s an upper bounds on where that matters, and I think it mostly matters when climbing the corporate ladder. If you look at the richest men in the world, most are not supermodels.
The moral of the story - if you want to play the game you have to play by the game’s rules. Otherwise, play a different game.