News for hackers. Stuff that matters.
Re:Twitter's garbage problem by sketerpot on Sunday May 16, 2010 @10:01PM (#1353273)
I can't believe nobody has mentioned naive Bayesian text classification yet. It sounds like it could work wonders for Twitter. I'm much more likely to be interested in tweets with words like "hylomorphism" than tweets with words like "omglol", and a text classification algorithm could learn that if you trained it up some. It doesn't have to be perfect; it just has to improve the signal-to-noise ratio significantly.
Re:Twitter's garbage problem by kmavm on Sunday May 16, 2010 @10:24PM (#1353313)
I've looked into this a bit, albeit more in a spam-filtering context; tweets have very little text for naive Bayes to latch onto. 140 characters would be 20-30 words, tops. That is so few words that it is hard to move the prior very much, unless there are blockbuster words that almost always indicate a bad tweet; as the article suggested, "breakfast", "beer", etc.
Re:Twitter's garbage problem by fizx on Monday May 17, 2010 @12:13AM (#1353532)
Here's a start. It's a simple ruby app that runs on heroku and proxies your twitter account, dropping uninteresting things. Extremely raw code.

http://github.com/fizx/sometweets

http://sometweets.heroku.com/

Re:Twitter's garbage problem by JCThoughtscream on Sunday May 16, 2010 @10:43PM (#1353353)
Filter by account, then, instead of by post? With a sample body of, say, the ten or twenty of an account's most recent posts.
Re:Twitter's garbage problem by kmavm on Monday May 17, 2010 @12:22AM (#1353553)
The article's whole premise was that tweet quality does not correlate well within an account; e.g., some marvelous twitter streams include breakfast tweets.
Re:Twitter's garbage problem by sketerpot on Sunday May 16, 2010 @10:54PM (#1353373)
As I said, I'm fine with a filter that has poor accuracy, as long as it makes twitter marginally better.
Re:Twitter's garbage problem by jshen on Sunday May 16, 2010 @10:43PM (#1353351)
Dont many classifiers pick the most interesting n words and only use those to decide? Where n is something like 15.
Re:Twitter's garbage problem by sketerpot on Sunday May 16, 2010 @10:54PM (#1353374)
They usually have a lot more than 15 words to choose from, though.
Re:Twitter's garbage problem by jshen on Monday May 17, 2010 @09:26PM (#1356117)
yeah, but I'm not sure it would make that much of a difference. I should throw together a classifier and see.
Re:Twitter's garbage problem by avk on Sunday May 16, 2010 @10:56PM (#1353375)
I'm currently solving this very problem with a much simpler algorithm and slowly letting folks into my beta over at http://slipstre.am/
Re:Twitter's garbage problem by retube on Monday May 17, 2010 @05:40AM (#1353924)
I think stat/learning classifiers could work, even with the brevity of tweets. However, as the author says, one persons gold is anothers garbage. You'd need a platform/system that would allow for personal classifiers for each/every tweet consumer.