Re:Fewer Than Half of Google Searches Now Result in a Click
by tylerl on Wednesday August 14, 2019 @11:03AM
(#20696268)
The bot doesn't need to crawl your site for it to be indexed; it crawls other sites that link to yours.
You can specify your index preferences in Webmaster Tools. Don't know if there's a domain-wide off switch in there, but there probably is.
Re:Fewer Than Half of Google Searches Now Result in a Click
by SahAssar on Wednesday August 14, 2019 @06:58PM
(#20700889)
Using Webmaster Tools is not a good option since it requires you register with the exact company you are probably trying to not interact with.
Re:Fewer Than Half of Google Searches Now Result in a Click
by jefftk on Wednesday August 14, 2019 @10:43AM
(#20696039)
The blog post you link has a bunch of alternatives, but I agree they're not great. If there are a lot of webmasters who want to be able to noindex through robots.txt then making the case for adding noindex to the standard would be a good next step.
(Still speaking only for myself)
Re:Fewer Than Half of Google Searches Now Result in a Click
by ademarre on Wednesday August 14, 2019 @11:18AM
(#20696430)
Googlebot actually used to support a noindex rule in robots.txt, but they are removing it.
https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
How do you tell Googlebot to not crawl your site and to not index it either?
Previously, one could use the undocumented "Noindex" directive in robots.txt, but this will be disabled soon: https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...