Re:Analyzing Articles on Hacker News Using NLP
by minimaxir on Wednesday August 01, 2018 @01:26AM
(#17659933)
> HN users/voters seem habituated and predictable. What if this is applied to link titles to predict likelihood of making it to the front page? What words (or sequences of letters) are associated with popularity?
Apropos of nothing, I am working on building a model for predicting HN post performance.
TL;DR it is not easy.
Re:Analyzing Articles on Hacker News Using NLP
by everdev on Wednesday August 01, 2018 @02:39AM
(#17660188)
This was built in '17:
https://intoli.com/blog/hacker-news-title-tool/
HN discussion: https://news.ycombinator.com/item?id=14400603
Re:Analyzing Articles on Hacker News Using NLP
by minimaxir on Wednesday August 01, 2018 @10:47AM
(#17662942)
Interesting and good to see that as a reference. However, I suspect that approach overfits, and the author does not mention using a validation set.
Re:Analyzing Articles on Hacker News Using NLP
by everdev on Wednesday August 01, 2018 @12:03PM
(#17663760)
Yes, apparently "Rust, Rust, Rust, Rust!" almost guarantees 1st page according to the model.
Re:Analyzing Articles on Hacker News Using NLP
by baccheion on Wednesday August 01, 2018 @01:37AM
(#17659970)
I did one some years ago (basic attempt using XGBoost) and AUC was high 0.6s or low 0.7s.
I think I treated the title as a string rather than words and each column/feature was one letter. That is, if there were 256 possible characters and each title could only be 80 characters long, there'd be 256 * 80 * 2 columns (title + title reversed).
Re:Analyzing Articles on Hacker News Using NLP
by minimaxir on Wednesday August 01, 2018 @01:57AM
(#17660053)
If you are doing a classification model, what’s the score threshold?
The trick with that approach is if you set the threshold high, it’s very easy to get a model with a high accuracy since classes will be imbalanced. (The vast, vast majority of HN submissions do not do well)
Re:Analyzing Articles on Hacker News Using NLP
by baccheion on Wednesday August 01, 2018 @02:25AM
(#17660151)
If I remember correctly, HN's data dump included a made_front_page flag (not exact name). Also, XGBoost includes a feature that attempts to normalize against imbalanced classes. As for thresholds, there's only including submissions with at least average +/- standard_deviation votes.
I also wonder how many unique voters are present, especially regulars and those who vote before a link hits the front page. I bet there aren't that many. And I bet they are mostly INTJ. That is, content seems curated/controlled by a handful of users (ie, bubble). What can be done to buffer against bias? How can submissions automatically be surfaced/tested (shown on the front page to patterned/known users) even before receiving any votes?
I've always thought some percentage of the front page should be dedicated to (randomly chosen, though slightly weighted) new submissions. Some percentage of the page some percentage of the time to some percentage of users able to vote. Or maybe the new tab should be shown inline on the right. That is, I'm guessing most only see what's shown on the first page.