>VGG-Face aims at representing a given face as a vector of scores that are as unaffected as possible by facial expression, background, lighting, head orientation, image properties such as brightness or contrast, and other factors that can vary across different images of the same person.
All the heavy lifting for this paper was done by logistic regression on the facial features reported by VGG-Face, they didn't train a DNN specifically for identifying sexuality from image pixel values. The algo wouldn't have seen clothing at all.
I still think that while the results are accurate(ish), they reveal fashion trends and not genetic changes. Can in point, the composite average of homosexual male has glasses. Facial features of homosexual are often gender atypical but that's trough a lot of effort to conform with fashion trends (eyebrow reshaping, smoother skin, "kinkier" smile, etc.). This is especially true on dating sites. The whole nature vs nurture thing.
While I agree that hormone levels can impact facial features, I have yet to see a strong study proving that hormones levels can be directly linked to sexual orientation or gender identity.
I am certain that you could introduce a lot of errors by adding a bunch of metrosexual but straight individuals into the mix.