News for hackers. Stuff that matters.

Ask PG: Polite ways to scrape HN?

Hi, I've been thinking about doing an HN related project and was wondering if the topic of crawling HN has been covered before. A search of "Ask PG" didn't bring up anything relevant nor did the guidelines and FAQ links. I'm also unaware of any api availability. There seems to be new HN side projects popping up every few weeks or so. How are you getting your HN data?

Obviously the basic rules apply, aka. don't hammer my server, duh. Assuming sequential article id's currently clocking in at ~1721000+ It would take a fare amount of time to slurp down HN even at 1k pages/day. Maybe look into pulling down the google cache. Eh, I dunno. Don't want to get any of my ip addrs banned or worse (hehe). Just looking for some guidance... Thanks in advance.

( 29 comments | 71 points | on HN )
Re:Ask PG: Polite ways to scrape HN? by pg on Thursday September 23, 2010 @07:33PM (#1721997)
You can crawl so long as you respect robots.txt and don't retrieve more than a couple pages a minute. (Conservative, I know, but we're serving 800k pages a day off one server, using an app written in a slow language.)
Re:Ask PG: Polite ways to scrape HN? by portman on Thursday September 23, 2010 @11:10PM (#1722497)
Paul, have you considered asking the community to volunteer resources?

You've mentioned several times that the site is on one server and that performance is an issue at times. I would donate time/money to make those problems go away, and I suspect others would as well.

Proposal: Post a new thread, "somebody build me a HN server farm", and include the software and hardware prerequisites for an Arc webserver. I bet within 48 hours you would have a 3-4 servers and a load balancer at your disposal.

Re:Ask PG: Polite ways to scrape HN? by jrockway on Thursday September 23, 2010 @11:13PM (#1722501)
I think the issue is that news.arc doesn't use a distributable storage system. It is as far from "shared nothing" as you get.
Re:Ask PG: Polite ways to scrape HN? by portman on Thursday September 23, 2010 @11:27PM (#1722548)
Hm, interesting.

You can scale anything, though. For example, a pair of reverse proxies in front of the single "share nothing" app server could reduce the load on that one machine. I'm sure the collective brilliance of the HN readership could come up with solutions.

Re:Ask PG: Polite ways to scrape HN? by staunch on Friday September 24, 2010 @12:04AM (#1722619)
Simple: buy a bigger server. It's amazing how powerful one machine can be these days.
Re:Ask PG: Polite ways to scrape HN? by grinich on Friday September 24, 2010 @04:52PM (#1725380)
Assuming nothing has changed, it's running on a "3.0 GHz Core whatever, 12 GB RAM, 64-bit FreeBSD 7.1."

Upgraded on 4/19/09 from a 2.4 GHz Pentium 4, 4 GB RAM, 32-bit FreeBSD 5.3.

rtm's comment: http://news.ycombinator.com/item?id=516122

Re:Ask PG: Polite ways to scrape HN? by jrockway on Friday September 24, 2010 @08:46PM (#1725960)
Yeah, well, this is a middle of the road server these days. I have a 16-core server with 64G of RAM as a dev server at work. They get even more powerful than this, too.

Not saying pg should spend any money on this, though. A redesign of news.arc will stretch the hardware a lot further.

Re:Ask PG: Polite ways to scrape HN? by siculars on Thursday September 23, 2010 @09:58PM (#1722304)
Thank you!
Re:Ask PG: Polite ways to scrape HN? by grails4life on Friday September 24, 2010 @05:11PM (#1725446)
Are you calling Arc Lisp slow?
Re:Ask PG: Polite ways to scrape HN? by bobds on Thursday September 23, 2010 @04:08PM (#1721164)
Here you go: http://api.ihackernews.com/

It's an unofficial API though and it might not do everything you need.

Re:Ask PG: Polite ways to scrape HN? by sh1mmer on Thursday September 23, 2010 @05:18PM (#1721502)
I also made a YQL table for HackerNews a while back which should be as polite to the site as possible.

http://developer.yahoo.com/yql/console/?q=select%20*%20from%...

YQL implements a bunch of caching layers so it should tread as lightly as possible.

Disclaimer: I do work for Yahoo!

Re:Ask PG: Polite ways to scrape HN? by util on Thursday September 23, 2010 @06:59PM (#1721895)
Just curious, especially after the deal with Microsoft: Has Yahoo made any long term commitments to support this? Would love to use this for some projects.
Re:Ask PG: Polite ways to scrape HN? by sh1mmer on Sunday September 26, 2010 @08:54AM (#1729065)
Yes. It's been explicitly stated on our blog and on the YQL page that it's not going away.

Specifically we've promised a minimum of 6mo notice if we ever did decide to shut it down.

Re:Ask PG: Polite ways to scrape HN? by dustingetz on Thursday September 23, 2010 @04:23PM (#1721259)
is this condoned by PG?
Re:Ask PG: Polite ways to scrape HN? by ronnier on Thursday September 23, 2010 @05:01PM (#1721431)
He's aware of it because he responded to an AskHN question about it: http://news.ycombinator.com/item?id=1702399

He didn't say not to do it :)

Re:Ask PG: Polite ways to scrape HN? by sahillavingia on Thursday September 23, 2010 @04:32PM (#1721310)
For now, the answer seems to be yes.
Re:Ask PG: Polite ways to scrape HN? by benatkin on Thursday September 23, 2010 @04:40PM (#1721345)
I'm inclined to agree. Remember, pg doesn't provide an RSS feed for his essays.

http://www.paulgraham.com/rss.html

Re:Ask PG: Polite ways to scrape HN? by riobard on Thursday September 23, 2010 @06:30PM (#1721800)
Any words on why there is no API for HN?
Re:Ask PG: Polite ways to scrape HN? by benatkin on Thursday September 23, 2010 @06:47PM (#1721848)
It might be a philosophical thing. One APIs became popular, scraping started to be viewed as breaking the rules. I think this has discouraged many from writing search engines. Crawlers are just scrapers on a big scale.

I know I'd like to see more serious attempts at vertical search engines. I wonder if getting backlash, in the form of people complaining to hosting companies, would be the hardest part of writing a search engine for ruby blog posts that understands regular expressions. I could certainly use such a tool.

Re:Ask PG: Polite ways to scrape HN? by sahillavingia on Thursday September 23, 2010 @11:23PM (#1722530)
Time that could be spent working on some other, higher-marginal-utility thing, I would think.
Re:Ask PG: Polite ways to scrape HN? by sahillavingia on Thursday September 23, 2010 @04:08PM (#1721163)
This works great: http://api.ihackernews.com/
Re:Ask PG: Polite ways to scrape HN? by lnp on Thursday September 23, 2010 @06:10PM (#1721710)
You can start with http://hotfile.com/dl/71319191/96579fe/ycombinator-news20080... - a mirror of old 'Y Combinator Dataset Of Posts Version 1.7' from http://news.ycombinator.com/item?id=296919 .

Also, could you please post final dataset somewhere when done?

Re:Ask PG: Polite ways to scrape HN? by ck2 on Friday September 24, 2010 @10:26AM (#1723721)
Why not crawl the google text cache instead?

It indexes HN within a few minutes of any post.

Re:Ask PG: Polite ways to scrape HN? by SteveMorin on Thursday September 23, 2010 @05:08PM (#1721466)
It's a good, question and api.ihackernews.com doesn't look bad
Re:Ask PG: Polite ways to scrape HN? by deutronium on Thursday September 23, 2010 @08:48PM (#1722150)
You could monitor http://hackerne.ws/newest so you wouldn't need spider to many pages, if you only wanted the latest content.
Re:Ask PG: Polite ways to scrape HN? by charlief on Thursday September 23, 2010 @07:55PM (#1722055)
Just curious, what kind of analysis are you thinking of doing?
Re:Ask PG: Polite ways to scrape HN? by jonpaul on Thursday September 23, 2010 @09:58PM (#1722303)
That's what interested me as well. What could you possibly want to scrape HN for? I suppose it would kind of fun to see who the most active users are. Or, who has the most karma. Maybe incorporate additional game mechanics?

The only site outside of HN that uses HN data that I use regularly is ihackernews.com. All of the others I have just registered and the sites lay dormant.

Re:Ask PG: Polite ways to scrape HN? by nitrogen on Friday September 24, 2010 @01:34AM (#1722820)
Some of what you want is here: http://news.ycombinator.com/lists
Re:Ask PG: Polite ways to scrape HN? by siculars on Thursday September 23, 2010 @10:02PM (#1722316)
If I ever get something worth looking at together I'll be sure to post here on HN. But I have some content analysis in mind.