News for hackers. Stuff that matters.
Re:Ask PG: Polite ways to scrape HN? by bobds on Thursday September 23, 2010 @04:08PM (#1721164)
Here you go: http://api.ihackernews.com/

It's an unofficial API though and it might not do everything you need.

Re:Ask PG: Polite ways to scrape HN? by sh1mmer on Thursday September 23, 2010 @05:18PM (#1721502)
I also made a YQL table for HackerNews a while back which should be as polite to the site as possible.

http://developer.yahoo.com/yql/console/?q=select%20*%20from%...

YQL implements a bunch of caching layers so it should tread as lightly as possible.

Disclaimer: I do work for Yahoo!

Re:Ask PG: Polite ways to scrape HN? by util on Thursday September 23, 2010 @06:59PM (#1721895)
Just curious, especially after the deal with Microsoft: Has Yahoo made any long term commitments to support this? Would love to use this for some projects.
Re:Ask PG: Polite ways to scrape HN? by sh1mmer on Sunday September 26, 2010 @08:54AM (#1729065)
Yes. It's been explicitly stated on our blog and on the YQL page that it's not going away.

Specifically we've promised a minimum of 6mo notice if we ever did decide to shut it down.

Re:Ask PG: Polite ways to scrape HN? by dustingetz on Thursday September 23, 2010 @04:23PM (#1721259)
is this condoned by PG?
Re:Ask PG: Polite ways to scrape HN? by ronnier on Thursday September 23, 2010 @05:01PM (#1721431)
He's aware of it because he responded to an AskHN question about it: http://news.ycombinator.com/item?id=1702399

He didn't say not to do it :)

Re:Ask PG: Polite ways to scrape HN? by sahillavingia on Thursday September 23, 2010 @04:32PM (#1721310)
For now, the answer seems to be yes.
Re:Ask PG: Polite ways to scrape HN? by benatkin on Thursday September 23, 2010 @04:40PM (#1721345)
I'm inclined to agree. Remember, pg doesn't provide an RSS feed for his essays.

http://www.paulgraham.com/rss.html

Re:Ask PG: Polite ways to scrape HN? by riobard on Thursday September 23, 2010 @06:30PM (#1721800)
Any words on why there is no API for HN?
Re:Ask PG: Polite ways to scrape HN? by benatkin on Thursday September 23, 2010 @06:47PM (#1721848)
It might be a philosophical thing. One APIs became popular, scraping started to be viewed as breaking the rules. I think this has discouraged many from writing search engines. Crawlers are just scrapers on a big scale.

I know I'd like to see more serious attempts at vertical search engines. I wonder if getting backlash, in the form of people complaining to hosting companies, would be the hardest part of writing a search engine for ruby blog posts that understands regular expressions. I could certainly use such a tool.

Re:Ask PG: Polite ways to scrape HN? by sahillavingia on Thursday September 23, 2010 @11:23PM (#1722530)
Time that could be spent working on some other, higher-marginal-utility thing, I would think.