Re:Linguistic Harbingers of Betrayal: A Case Study on an Onl...
by philh on Tuesday August 11, 2015 @08:17AM
(#10040405)
Unsurprisingly, that 57% is almost meaningless.
It looks like they took all the betrayals they had, then matched each betrayal with a similar friendship that didn't end in betrayal, and somehow ended up with a data set that was about 52% lasting friendships and 48% betrayals. I think that "somehow" is that each friendship added a data point per season, and lasting friendships lasted longer than eventual-betrayal friendships.
Within that dataset, they got 57% accuracy on the question "will this friendship end in a betrayal?" That doesn't mean they can look at an arbitrary existing friendship and predict betrayal with 57% accuracy, because the base rates will be different. And in any case, it's important to distinguish between false positives and false negatives, which the accuracy score completely fails to do.
But they also looked at eventual-betrayal friendships, and attempted to predict when the betrayal would happen. (Or at least "will it happen this turn?") The base rate for that was 14% and they got an F1 score (https://en.wikipedia.org/wiki/F1_score) of 0.31, which I think is decent but not fantastic.
Re:Linguistic Harbingers of Betrayal: A Case Study on an Onl...
by jmnicolas on Tuesday August 11, 2015 @08:16AM
(#10040402)
Look at the second comment from the article (from Matt) :
"[...]and you use this computer algorithm to predict betrayal, the computer will predict that you'll be betrayed 4 times during those 10 turns. That's a lot of false positives[...]"
If I read it correctly, that's a biggie, especially given the weak outperformance of chance.
It looks like they took all the betrayals they had, then matched each betrayal with a similar friendship that didn't end in betrayal, and somehow ended up with a data set that was about 52% lasting friendships and 48% betrayals. I think that "somehow" is that each friendship added a data point per season, and lasting friendships lasted longer than eventual-betrayal friendships.
Within that dataset, they got 57% accuracy on the question "will this friendship end in a betrayal?" That doesn't mean they can look at an arbitrary existing friendship and predict betrayal with 57% accuracy, because the base rates will be different. And in any case, it's important to distinguish between false positives and false negatives, which the accuracy score completely fails to do.
But they also looked at eventual-betrayal friendships, and attempted to predict when the betrayal would happen. (Or at least "will it happen this turn?") The base rate for that was 14% and they got an F1 score (https://en.wikipedia.org/wiki/F1_score) of 0.31, which I think is decent but not fantastic.
And what happens if one combines that data with other data?
"[...]and you use this computer algorithm to predict betrayal, the computer will predict that you'll be betrayed 4 times during those 10 turns. That's a lot of false positives[...]"