Re:Ask PG: Polite ways to scrape HN?
by sahillavingia on Thursday September 23, 2010 @04:32PM
(#1721310)
For now, the answer seems to be yes.
[ Parent
| on HN ]
I'm inclined to agree. Remember, pg doesn't provide an RSS feed for his essays.
Any words on why there is no API for HN?
It might be a philosophical thing. One APIs became popular, scraping started to be viewed as breaking the rules. I think this has discouraged many from writing search engines. Crawlers are just scrapers on a big scale.
I know I'd like to see more serious attempts at vertical search engines. I wonder if getting backlash, in the form of people complaining to hosting companies, would be the hardest part of writing a search engine for ruby blog posts that understands regular expressions. I could certainly use such a tool.
Re:Ask PG: Polite ways to scrape HN?
by sahillavingia on Thursday September 23, 2010 @11:23PM
(#1722530)
Time that could be spent working on some other, higher-marginal-utility thing, I would think.