Because even when you find nice clothes the sizing is often difficult. Thrift shops are often like a gold mine (I do recognize that normal shops often have similar problems depending on one's taste). Not everyone wants to mine.

Visibly damaged clothes are not good for every occasion and may just look bad on almost every occasion. Otherwise why not to wear a potato sack or rags tied with a bit of string?