News for hackers. Stuff that matters.

A comment on Google’s robots.txt parser is now open source. Parent comment

Re:Google’s robots.txt parser is now open source by knorker on Monday July 01, 2019 @11:31AM (#20326345)
I'm not surprised. Some people think humans read robots.txt and get super angry when the crawler doesn't understand.
[ Parent | on HN ]
Re:Google’s robots.txt parser is now open source by c3534l on Monday July 01, 2019 @11:36AM (#20326401)
I read robots.txt, but I'm not a massive corporation.
[ Parent | on HN ]