News for hackers. Stuff that matters.
Re:Fewer Than Half of Google Searches Now Result in a Click by areyousure on Wednesday August 14, 2019 @10:17AM (#20695786)
But that requires that Googlebot be allowed to crawl the page in robots.txt in the first place.

How do you tell Googlebot to not crawl your site and to not index it either?

Previously, one could use the undocumented "Noindex" directive in robots.txt, but this will be disabled soon: https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...

Re:Fewer Than Half of Google Searches Now Result in a Click by tylerl on Wednesday August 14, 2019 @11:03AM (#20696268)
The bot doesn't need to crawl your site for it to be indexed; it crawls other sites that link to yours.

You can specify your index preferences in Webmaster Tools. Don't know if there's a domain-wide off switch in there, but there probably is.

Re:Fewer Than Half of Google Searches Now Result in a Click by SahAssar on Wednesday August 14, 2019 @06:58PM (#20700889)
Using Webmaster Tools is not a good option since it requires you register with the exact company you are probably trying to not interact with.
Re:Fewer Than Half of Google Searches Now Result in a Click by jefftk on Wednesday August 14, 2019 @10:43AM (#20696039)
The blog post you link has a bunch of alternatives, but I agree they're not great. If there are a lot of webmasters who want to be able to noindex through robots.txt then making the case for adding noindex to the standard would be a good next step.

(Still speaking only for myself)

Re:Fewer Than Half of Google Searches Now Result in a Click by ademarre on Wednesday August 14, 2019 @11:18AM (#20696430)
Googlebot actually used to support a noindex rule in robots.txt, but they are removing it.

https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...

Re:Fewer Than Half of Google Searches Now Result in a Click by jefftk on Wednesday August 14, 2019 @12:43PM (#20697288)
Yes, that was linked above. It looks like this is part of reducing support to what's in the spec?
Re:Fewer Than Half of Google Searches Now Result in a Click by ademarre on Thursday August 15, 2019 @03:45AM (#20703367)
Oops, yep. I didn't see that context.