News for hackers. Stuff that matters.
Re:Search Goes Real-Time With Scoopler (YC S08) by okeumeni on Friday May 08, 2009 @11:25AM (#599739)
I have one question, are they consuming other companies search API, let’s say Twitter or are they crawling and aggregate their own data? Either way it’s a great job!
Re:Search Goes Real-Time With Scoopler (YC S08) by ajasver on Friday May 08, 2009 @11:32AM (#599750)
We actually crawl and index all the data ourselves. That's how we identify the most popular content.
Re:Search Goes Real-Time With Scoopler (YC S08) by hachiya on Friday May 08, 2009 @12:44PM (#599874)
Is this truly crawling the sites, or querying each site's search service?

e.g. for Twitter, are you crawling all the Twitter users' URLs (seems difficult to do - how do you find new accounts' URLs?), or are you fetching results from search.twitter.com ?

Re:Search Goes Real-Time With Scoopler (YC S08) by ajasver on Friday May 08, 2009 @03:45PM (#600140)
We take the raw data streams (e.g. a data mining feed from Twitter) and process them, ourselves.
Re:Search Goes Real-Time With Scoopler (YC S08) by dannyr on Friday May 08, 2009 @03:01PM (#600080)
Yeah. I'm curious if they developed something similar to Twitter's search functionality or they are using Twitter's Trends feed.

In its current form, most of the results are from Twitter which doesn't really differentiate it much from Twitter Search.

Re:Search Goes Real-Time With Scoopler (YC S08) by zaidf on Friday May 08, 2009 @02:11PM (#600002)
I don't think crawling and querying are mutually exclusive ideas.
Re:Search Goes Real-Time With Scoopler (YC S08) by okeumeni on Friday May 08, 2009 @11:41AM (#599767)
Nice Work!