News for hackers. Stuff that matters.
Re:Google’s robots.txt parser is now open source by pedrorijo91 on Monday July 01, 2019 @03:17PM (#20328765)
can this been seen as a initiative to make google robots.txt parser the internet standard? every webmaster will want to be compliant with google corner cases...
Re:Google’s robots.txt parser is now open source by H8crilA on Monday July 01, 2019 @06:39PM (#20330425)
That's probably the hidden agenda.
Re:Google’s robots.txt parser is now open source by djsavvy on Monday July 01, 2019 @09:36PM (#20331381)
It's not even hidden, Google explicitly says that in the blog post.