It's a counter-intuitive argument, but an extremely astute one. Consider: Many people would rather not wear an ill-fitting 'nice' outfit, or may not have the sartorial education to do so, but feel obliged to do so as a condition of their employment. You won't be able to get a job as a bank clerk if you show up dressed in a hoodie and jeans, for example; you need to show up in a shirt and tie (if you're a guy) even it's the cheapest item that you could find at Men's Wearhouse and you don't look very good in it.

'Well-fitting slob clothing' not only signals that you're not subject to some sort of onerous dress code imposed by your employer (within reason), but also that you can afford to spend money on what most people could consider leisurewear. Further, while you may genuinely not be interested in clothes, picking them out in the morning, looking at other peoples', or shopping for them - and while I can sympathize with your disinterest - that very indifference is itself a mark of affluence. Most people work hard at that stuff because they feel obliged to create a certain image to get employed, get laid, and maintain their self-respect. If they don't care about clothes and they choose not to spend money on them, and they don't happen to put out certain other socioeconomic signals, then they will just be dismissed as scruffy bums.