That's something I think could be automated (at least for me.)
I can (and do) guess whether I am likely to be interested in a new programming language, based on what other languages the commenters compare it to.
In my case it would probably be something like:
* Compared to Haskell, Io or SETL => very interesting
* Compared to Java, JS or Scheme => neutral
* Compared to Groovy, PHP and VB => uninteresting
What makes you think that a Naive Bayes classifier will automatically put all unseen/unexpected/surprising data in just one of two categories?
Indeed, to the classifier it's just two categories, it doesn't "know" what's "in" or "out".
In fact it is much more likely that the classifier will sort of distribute the data that is unexpected (read: doesn't contain many features that are trained on) rather evenly among the two categories based on other features. Which is exactly what you would want it to do.
The author also said he plotted the precision-recall curves? (too bad he just showed a screenshot with the numbers instead of the graphs) That sort of analysis is bound to bring out such behaviour.
HN's subtitle is "Links for the intellectually curious"
I guess HN has a dual purpose to keep people up on "breaking" hacker news, but I like to think of it as "hacker news outside your thinking pattern".
Also why do people immediately go to AWS for testing something? Doesn't a real hacker have their own server handy for experimental projects, or is it only me?