Re:De-anonymizing programmers from executable binaries
by IncRnd on Friday March 16, 2018 @03:46PM
(#16603502)
Great point. A system needs to be forward tested not simply back tested on a known set of data.
Incorrect back testing is absolutely standard with stock/bond/forex/etc trading. There are a million hustlers who sell their back-tested systems. In essence, without forward testing all you have is an untested hypothesis.
I found it was easy to get mind-blowing results on a single dataset - e.g. 99% accuracy for working out who wrote which emails in the Enron [0] dataset.
The moment the test set was even slightly cross-genre, results became bad.
So this is great in theory, but I would love to see a test set from the same authors outside the context of the Google Code Jam. I'd be surprised if the results were anything like as good.
There are very few real-world cases where you have a large amount of same-genre data for the questioned author. (No one writes a thousand suicide notes / ransom demands / scams)
[0] A large email database often used for authorship tasks https://www.cs.cmu.edu/~enron/