News for hackers. Stuff that matters.

A comment on Google’s robots.txt parser is now open source. Parent comment

Re:Google’s robots.txt parser is now open source by kortilla on Monday July 01, 2019 @12:47PM (#20327189)
If you don’t want something public, don’t allow a crawler to find it or access it. The people you want to hide stuff from are just going to use search engines that ignore robots.txt
[ Parent | on HN ]
Re:Google’s robots.txt parser is now open source by bhartzer on Monday July 01, 2019 @01:36PM (#20327721)
If you don't want someone or a bot to find it, don't put it online.
[ Parent | on HN ]