News for hackers. Stuff that matters.
Re:The “Chad” bug by Animats on Friday January 08, 2016 @04:08AM (#10863757)
The strict interpretation is that if "robots.txt" returns 403 Forbidden, it's interpreted as "deny all". That's what the Python library does. We list those sites as "Blocked".

There are no comments on this comment yet.