Re:The “Chad” bug
by Animats on Friday January 08, 2016 @04:08AM
(#10863757)
The strict interpretation is that if "robots.txt" returns 403 Forbidden, it's interpreted as "deny all". That's what the Python library does. We list those sites as "Blocked".
[ Parent
| on HN ]
There are no comments on this comment yet.