News for hackers. Stuff that matters.

A comment on Any good api to scrape HN other than this?. Parent comment

Re:Any good api to scrape HN other than this? by nashequilibrium on Monday December 23, 2013 @06:21AM (#6953848)
rp = robotparser.RobotFileParser()

rp.set_url("https://news.ycombinator.com/news/robots.txt")

rp.read()

# Reads the robots.txt

rp.can_fetch("*", 'https://news.ycombinator.com/news')

>>>> True

[ Parent | on HN ]
Re:Any good api to scrape HN other than this? by kaushikfrnd on Monday December 23, 2013 @07:03AM (#6953919)
cool
[ Parent | on HN ]