Re:Ask PG: Polite ways to scrape HN?
by dustingetz on Thursday September 23, 2010 @04:23PM
(#1721259)
is this condoned by PG?
[ Parent
| on HN ]
He's aware of it because he responded to an AskHN question about it: http://news.ycombinator.com/item?id=1702399
He didn't say not to do it :)
Re:Ask PG: Polite ways to scrape HN?
by sahillavingia on Thursday September 23, 2010 @04:32PM
(#1721310)
For now, the answer seems to be yes.
I'm inclined to agree. Remember, pg doesn't provide an RSS feed for his essays.
Any words on why there is no API for HN?
It might be a philosophical thing. One APIs became popular, scraping started to be viewed as breaking the rules. I think this has discouraged many from writing search engines. Crawlers are just scrapers on a big scale.
I know I'd like to see more serious attempts at vertical search engines. I wonder if getting backlash, in the form of people complaining to hosting companies, would be the hardest part of writing a search engine for ruby blog posts that understands regular expressions. I could certainly use such a tool.
Re:Ask PG: Polite ways to scrape HN?
by sahillavingia on Thursday September 23, 2010 @11:23PM
(#1722530)
Time that could be spent working on some other, higher-marginal-utility thing, I would think.