Image is one of those laws of society that we all wish wasn't true, all believe shouldn't be true, but is intractable and almost certainly derived from evolutionary pressure.

"Vestis virum facit", or clothes make the man is extraordinarily powerful in business and social dealings. Clothes signal a lot of information quickly. So this isn't about you caring what other people think, it's about how what other people care about affect your options. I could prescribe a few experiments to witness the differences directly but I'll tell you from personal experience that the A-B differences are stark.

There was a time when I was younger that I raged against this rule, but it's a youthful variation of "old man yells at cloud". You can try and create a new reality in your head and believe it fervently, and cut your nose to spite your face, but image matters greatly to other people, at a deep level.