My wife and I have discussed this before and never come to a satisfying conclusion. Best brief answer I've got is that the phenomena is rooted in historical/cultural uses of clothes, involving practicality/conformity for men, and more ornamentation for women (not drawing a judgement here, just considering the last 100-200 years in Western culture).