News for hackers. Stuff that matters.
Re:HackerFollow: Follow HN Users by erickerr on Wednesday October 06, 2010 @11:21AM (#1764283)
We'd love to do that but the data isn't readily available by scraping. We could continuously fetch the parent_id post until we reach the root story, but that would result in a lot of extra curl requests.. V2 : )
Re:HackerFollow: Follow HN Users by mahmud on Wednesday October 06, 2010 @11:26AM (#1764308)
Instead of crawling by #id, crawl by new posts from the /newest page. For each post, split it into multiple pages, setting the parent id/title that way. Not that you have to, but a future suggestion.
Re:HackerFollow: Follow HN Users by jeromec on Wednesday October 06, 2010 @11:42AM (#1764394)
I'm guessing the info is currently scraped from the 'threads?id=username' page, but the title to each story is already there after the word 'on'.
Re:HackerFollow: Follow HN Users by dvk on Wednesday October 06, 2010 @12:55PM (#1764763)
see: http://github.com/dkeskar/hnbot

/threads is not polite to robots.txt, so discussion.rb is deprecated.