News for hackers. Stuff that matters.

A comment on Google’s robots.txt parser is now open source. Parent comment

Re:Google’s robots.txt parser is now open source by adrianmonk on Monday July 01, 2019 @02:34PM (#20328326)
> evidence to me that this Noindex thing is bullshit

For those who (like me) don't know a lot about this, which side of the argument is bullshit? Have you just been proved right or wrong?

[ Parent | on HN ]
Re:Google’s robots.txt parser is now open source by jxcl on Monday July 01, 2019 @03:45PM (#20329028)
It looks like it's too late for me to edit my comment, but I've been proved right. Putting a Noindex directive directly in robots.txt is frequently suggested, but this seems like definitive proof that that does nothing (at least with Google).

As far as I can tell the inception of this idea was that it was briefly mentioned by some Google employee in an interview. Maybe it was supported in the past or maybe he just misspoke, but I bet even now we'll see people still using this tag.

[ Parent | on HN ]
Re:Google’s robots.txt parser is now open source by klohto on Monday July 01, 2019 @02:39PM (#20328360)
I recommend finishing the whole comment
[ Parent | on HN ]
Re:Google’s robots.txt parser is now open source by celeritascelery on Monday July 01, 2019 @03:42PM (#20328988)
read the whole comment. Still confused as well.
[ Parent | on HN ]