News for hackers. Stuff that matters.

A comment on The “Chad” bug. Parent comment

Re:The “Chad” bug by chris_wot on Friday January 01, 2016 @11:38AM (#10822285)
I'd assume they want you to crawl their website. When they say you are ignoring their robots.txt file, tell them you actively prevented you from seeing it and you could only assume that mean they WANTED to be crawled.

That would get them to fix the issue pretty quickly :-)

[ Parent | on HN ]
Re:The “Chad” bug by rdancer on Friday January 01, 2016 @06:30PM (#10824086)
No `robots.txt` indeed means weapons free, but crawler gleans useful info from it, e.g. which areas of the site are dynamically generated.

More likely the site is trying to serve custom versions of `robots.txt` to different bots, with good intentions, and the code is buggy.

[ Parent | on HN ]
Re:The “Chad” bug by Animats on Friday January 08, 2016 @04:08AM (#10863757)
The strict interpretation is that if "robots.txt" returns 403 Forbidden, it's interpreted as "deny all". That's what the Python library does. We list those sites as "Blocked".
[ Parent | on HN ]