News for hackers. Stuff that matters.
Re:Watch us crawl the news in realtime by jbr on Wednesday August 18, 2010 @06:13PM (#1615794)
Sure. This probably will turn into a blog entry, but here's the gist:

  4 medium sized linodes, divided as follows:
  1 app:    unicorn (rails), nginx
  1 db:     redis, solr, mysql
  2 worker: resque workers (both ruby/rails node.js) & the crawler
The crawler is written in node.js, backed by redis. When it finds a new page, it downloads it to shared local storage and adds a task to a resque queue monitored by the rails workers. They add a row to a mysql table that represents the permanent record of the page, use nokogiri to extract the body content and any metadata, index it into solr, delete the local copy, and upload the page to an s3 archive. When you request the page, rails asks solr.
Re:Watch us crawl the news in realtime by nolite on Wednesday August 18, 2010 @07:47PM (#1616004)
Nice, thanks..any stats on how fast it is? It looks pretty fast from the web page update
Re:Watch us crawl the news in realtime by jbr on Wednesday August 18, 2010 @08:53PM (#1616124)
Haven't built in much monitoring yet, but watching the resque web interface, most of the delay is in actually finding the new page - from there to it showing up on your screen is no more than a second or two. For subscribed users, we do email notifications and we almost always beat google news alerts, often by around 15 minutes.
Re:Watch us crawl the news in realtime by mikecane on Sunday August 22, 2010 @06:34PM (#1625413)
I have it bookmarked and find myself peeking in at least once a day during other news downtime.