News for hackers. Stuff that matters.
Re:A new algorithm predicts which Twitter topics will trend ... by denzil_correa on Wednesday November 07, 2012 @09:34AM (#4753491)
The only concern I have is the way the experiment is conducted. The correct way to collect data would be to consume the Streaming API to collect the random sample of tweets provided by Twitter [0]. They should then run their algorithm to compare how they are doing against Twitter's algorithm. Currently, they are filtering out certain topics of interest and just comparing these topics. Therefore, they are filtering a great degree of noise which is actually observed by Twitter.

[0] https://dev.twitter.com/docs/api/1.1/get/statuses/sample

Re:A new algorithm predicts which Twitter topics will trend ... by snikolov on Wednesday November 07, 2012 @12:41PM (#4754297)
Thanks for the interest! We collected tweets by sampling a small percentage of all tweets in a time window, to emulate what one might get from the streaming API (I did this as part of the VI-A http://vi-a.mit.edu/ masters program at MIT, and was an employee of Twitter). We did pick a fixed set of topics to track, but those were randomly sampled (though we did get rid of topics that trended multiple times in a large time window, like the name of a football player who scored in multiple matches, in which case we don't know which event we are trying to detect). One thing the algorithm doesn't do at the moment is come up with its own trending topics. It just tests prediction of trending/non-trending on a hold-out set taken from the original set of topics.