News for hackers. Stuff that matters.

A comment on Tusk – Short-lived anonymous feedback. Parent comment

Re:Tusk – Short-lived anonymous feedback by mxxx on Monday February 08, 2021 @06:28AM (#26063192)
Unless I’m mistaken, google don’t actually respect robots.txt any more. They recommend the use of some meta tag instead, from memory.
[ Parent | on HN ]
Re:Tusk – Short-lived anonymous feedback by yarcob on Monday February 08, 2021 @09:09AM (#26064634)
Their docs say they do: https://developers.google.com/search/reference/robots_txt

They do say that they ignore robots.txt for user-initiated actions eg. Google Translate, which makes sense in my opinion.

[ Parent | on HN ]
Re:Tusk – Short-lived anonymous feedback by mxxx on Monday February 08, 2021 @03:14PM (#26069522)
Ah yes you’re right, my apologies. They don’t support the noindex directive inside a robots.txt any more.
[ Parent | on HN ]