News for hackers. Stuff that matters.
Re:The island where people forget to die by pmelendez on Wednesday October 24, 2012 @01:23PM (#4693961)
> "If there is one thing I wish I could do to improve HN it would be to detect this sort of middlebrow dismissal algorithmically."

Do you have some data that we could use to play with? That sounds like a nice problem to solve.

Re:The island where people forget to die by pg on Wednesday October 24, 2012 @01:27PM (#4693981)
You already have the data I'd use: the text of the comments.

If anyone wants to try to train a filter to detect this sort of comment, I'd be very interested to see the result.

Re:The island where people forget to die by sakai on Wednesday October 24, 2012 @01:56PM (#4694124)
I'll consider that a challenge...

To any that have experience getting comments data from HN -- what's the fastest, most polite way to do this? And am I correct in remembering that there's some aggressive rate-limiting for crawling the site?

Re:The island where people forget to die by pg on Wednesday October 24, 2012 @02:30PM (#4694308)
Don't crawl the site, please. The place to get data is the HNSearch API.
Re:The island where people forget to die by ippisl on Wednesday October 24, 2012 @04:58PM (#4695019)
There's a database(quite old) of HN posts and comments here:

http://www.btscene.eu/details/2240774/Hacker+News+Database+o...

Re:The island where people forget to die by malandrew on Wednesday October 24, 2012 @05:06PM (#4695060)
Any plans to try to get a dataset for supervised ml? Perhaps collect the top 4 comments from all front page threads and post a survey on HN asking HNers to rate those comments for skepticism/dismissiveness?
Re:The island where people forget to die by derefr on Wednesday October 24, 2012 @07:08PM (#4695488)
I might bet that upvotes to middlebrow dismissal would be highly correlated with downvotes to the article itself, if we had downvotes to articles. In fact, I believe that these comments rise to the top because people are downvoting the article vicariously, by upvoting a rebuttal, however vacuous. In order to confirm this hypothesis, though, you'd have to collect data by implementing a downvote button on articles -- though it would not necessarily have to do anything in article-ranking terms ;)

(That might introduce a confounding factor, though--namely that by alleviating people's urge to downvote the article by giving them a [nonfunctional] button to do just that, people might stop upvoting the dismissive comments. Hmm....)

Re:The island where people forget to die by hntester123 on Wednesday October 24, 2012 @01:44PM (#4694063)
>You already have the data I'd use: the text of the comments.

Wouldn't that require real AI though? I thought for a minute that NLP (Natural Language Processing, not the other meaning(s) of the acronym) might help, but then thought that it may not work for cases where the comment is quoting another comment. Note: I'm not at all an expert in any of those fields, just interested.

Re:The island where people forget to die by adimitrov on Wednesday October 24, 2012 @02:00PM (#4694137)
Sounds like a job for Sentiment Analysis [1]. Modern systems are pretty good at discerning negative from positive comments.

You could probably find a way to mark negative and positive comments. Whether the resulting algorithm would be fine-grained enough to semi-reliably mark 'middlebrow dismissal,' I really don't know. Actually, as somebody who has worked on that stuff in the past, I don't think it would be very easy.

[1] http://en.wikipedia.org/wiki/Sentiment_analysis

Re:The island where people forget to die by pmelendez on Wednesday October 24, 2012 @02:17PM (#4694245)
Thanks for the link.

Agreed I don't think it would be an easy task, but I wonder how would perform a "bag of word" approach.

Harder part as I see it would be to categorize the comments on middlebrow dismissal / Not dismissal. It seems like we would be spending more time preparing the data than in the algorithm itself.

Re:The island where people forget to die by disgruntledphd2 on Wednesday October 24, 2012 @04:09PM (#4694809)
Welcome to statistics/data science/machine learning.

The hardest part is always getting the data into usable form. Its not as much fun as fitting models, but its definitely the majority of any role where people pay you to do this kind of stuff.

There's a lot of good research on forums (pm me if you want a bibliography i collected for a previous role), and short texts have become a bigger deal post Twitter. I completely agree with pg on the somewhat annoying nature of comments such as the GP.

Re:The island where people forget to die by pmelendez on Wednesday October 24, 2012 @05:00PM (#4695033)
Hey... Thanks for the offer! How can I contact you? I didn't know we can pm users over here.

I did have worked with ML before but mostly with images which are (IMHO) way easier to put in a format depending on the problem.

Re:The island where people forget to die by disgruntledphd2 on Friday October 26, 2012 @01:51PM (#4703343)
My email is obsfuscated in my profile. Should be pretty clear, conditional on your humanity.
Re:The island where people forget to die by benvanderbeek on Wednesday October 24, 2012 @03:43PM (#4694677)
Bing Liu is one of the researchers working on this. I've discussed Amazon review fraud detection with him. http://www.cs.uic.edu/~liub/
Re:The island where people forget to die by hntester123 on Wednesday October 24, 2012 @02:15PM (#4694235)
Hadn't heard of it; thanks for the link.
Re:The island where people forget to die by heed on Wednesday October 24, 2012 @02:06PM (#4694173)
It might be easier to investigate submissions first and filter at this level if a trend is discovered. That is, a certain type of submission might attract a certain attitude of comment.