Look at sex differences in animals. There are many, many, cases where one sex does the searching for (colorful, lots of adornment) and the other is the searched-for (drab, no adornment).

It is entirely reasonable to assume that there is a biological basis for the sex differences in men and womens fashion. There may not be proof that the sex differences are innate, but it's reasonable to propose that they are innate.

By the same token, it is unreasonable to deny that these sex differences are innate, until such time as proof is offered.

> I have to say I'm quite suspicious of inherent genetic traits that only reveal themselves when the textile industry appears

There's also every reason to believe that people were modifying their appearance before textiles were around. See scarification, use of colored clays / paints, etc. Even something like singing can be used as a way to attract mates. There's no reason to pick textiles as the sole use of fashion.