News for hackers. Stuff that matters.
Re:Show HN: Hacker News with categories and popularity by federkasten on Wednesday February 05, 2014 @08:55PM (#7188238)
Using natural language processing and supervised learning, it automatically tagged news with 9 categories.
Re:Show HN: Hacker News with categories and popularity by alixaxel on Wednesday February 05, 2014 @09:42PM (#7188403)
Should http://news.ycombinator.com/item?id=7183076 be categorized as Web Technology then?
Re:Show HN: Hacker News with categories and popularity by Houshalter on Thursday February 06, 2014 @04:54AM (#7189550)
I wonder if you clustered HN articles to find categories automatically. That may not be useful, but I'd be interested in seeing how HN articles are naturally categorized.
Re:Show HN: Hacker News with categories and popularity by xux on Wednesday February 05, 2014 @09:13PM (#7188304)
That's really interesting. Will you open-source the NLP algorithm?
Re:Show HN: Hacker News with categories and popularity by federkasten on Wednesday February 05, 2014 @09:41PM (#7188400)
I am planning to open source it in several months. (Our codes have not been well-commented and well-structured yet...

Our implementation and algorithm detail is followings.

Its categorizing process is written in Python.

Using nltk, it makes corpus with TFIDF model from HN topics and comments. And it generates classifiers from this corpus with SVM algorithm using scipy and numpy.

FYI, its web interface is written in Clojure and ClojureScript.

Re:Show HN: Hacker News with categories and popularity by hnriot on Thursday February 06, 2014 @01:26AM (#7189024)
presumably you've trained it with hand annotated content, or bootstrapped from a few choice hn searches (like ?q=jquery will give you a web tech category)
Re:Show HN: Hacker News with categories and popularity by federkasten on Thursday February 06, 2014 @04:00AM (#7189394)
Yes. You are right.

I trained classifiers with hand annotations (about 1000 contents or so)