The robotparser library works fairly well. When I used it, it choked on a particular not-to-spec robots.txt file, but other than that it was fine.

Copypasta of dead comment:

nashequilibrium 13 hours ago | link [dead]

rp = robotparser.RobotFileParser()

rp.set_url("https://news.ycombinator.com/news/robots.txt")

rp.read()

# Reads the robots.txt

rp.can_fetch("*", 'https://news.ycombinator.com/news')

>>>> True