News for hackers. Stuff that matters.

A comment on Fast scraping in Python with Asyncio.

Re:Fast scraping in Python with Asyncio by preinheimer on Monday March 03, 2014 @08:19PM (#7337838)
I'd really like it if articles on scraping included robots.txt support.
[ Parent | on HN ]
Re:Fast scraping in Python with Asyncio by NoodleIncident on Tuesday March 04, 2014 @09:58AM (#7340527)
The robotparser library works fairly well. When I used it, it choked on a particular not-to-spec robots.txt file, but other than that it was fine.

Copypasta of dead comment:

nashequilibrium 13 hours ago | link [dead]

rp = robotparser.RobotFileParser()

rp.set_url("https://news.ycombinator.com/news/robots.txt")

rp.read()

# Reads the robots.txt

rp.can_fetch("*", 'https://news.ycombinator.com/news')

>>>> True

[ Parent | on HN ]