True, but they also validated it against generic facebook images; and compared the accuracy of humans on the (dating website) images to the established benchmark of human capability (which came out comparable). If there was something telling sexuality-wise from the dating pictures, then you would probably expect humans to be able to pick up on it as well.

(see author notes on this: https://docs.google.com/document/d/11oGZ1Ke3wK9E3BtOFfGfUQuu...)

Maybe not but still impressive that they beat humans, no?

Well, assuming the data isn't good — no, obviouly. What "beating humans" would even mean in that case? Imagine you have a dataset of dog and squirrel pictures, with "dog" and "squirrel" (or, actually, "elephant" and "shark") labels assigned quasi-randomly. You make humans and NNs to compete in what label you assigned to them. NN beats humans. Now, what the fuck that result even means?

yeah but I guess the data isn't bad in that way in that the sexual orientation of people on the site is probably labeled accurately (in fact I would expect there it's more accurate than, you don't want to lie about what you want on the dating service), the data is bad in that it is perhaps not a representative sample of the population as a whole?

What do you mean by labelled accurately? The AI only predicts how an individual self-labelled on a fairly public dating service, not how this person feels or behaves in any other way. There are so many headless torsos on Grindr (or any other gay hookup app) because many men don't want to self label as gay when they look for sex with other men. These men could very well be looking for women on Tinder.

ok, I hadn't considered that.

> is probably labeled accurately

> the data is bad in that it is perhaps not a representative sample of the population as a whole?

Yeah that's my thinking too

that made me realize,

if you had a bad data set and let the computer train beforehand with the learning data, the human will need to "retrain" as they go (eventually realizing past stereotypes are failing in this atypical data set) and of course are going to do shit.

In a fair test both humans and computers would get to study the learning data.

Lots of AI journalism refers to "beating humans" but that's a hopelessly vague and unhelpful comparison. No AI can ever beat a human if by "human" you mean a human equipped with whatever tools the human may choose to use, because those tools would include the AI against which you would have it compete. So the comparison only makes sense if you restrict the human in some way. Depending on how you restrict the human, beating the human may be very easy or very hard. For example, if you insist that the decision be made within 100 ms then beating the human is very easy.

I think the usual way in which these experiments fiddle things so that the AI wins is by giving the AI lots of training data but giving the humans no training whatsoever, which may seem grossly unfair, but apparently that's acceptable in AI research/journalism.

Me: Horses can run faster than people!

You: "Faster than people"?! That's a hopelessly vague and unhelpful comparison. No horse can ever go faster than a human if by "human" you mean a human equipped with whatever tools the human may choose to use, because those tools would include the horse against which you would have it compete. So the comparison only makes sense if you restrict the human in some way.

Me: Um..

:-) The other stuff you say makes perfect sense, but that first bit is more pedantic than fair, I think.

You make a good point. There are many situations in which it's fairly obvious how the comparison ought to be made: the human should not be allowed to use a horse in competing with the horse in "running", and should not be allowed to use a computer in competing with the computer in "guessing sexuality". However, if the competition is in "predicting share price movements" then surely it would be strange not to allow some kind of computational assistance for the human, so in those cases, at least, it gets a bit hard to decide where to draw the line.

A bit unrelated but there are horse vs people marathons. and many times people win. that is because horses cannot cool themselves like people can

Yes, see "persistence hunting". It turns out that although humans may be thrashed by a poodle over short distances they are really rather good at extreme-long-distance races.