Re:Removing my site from Google search
by ghassanmas on Sunday October 03, 2021 @04:27PM
(#28740268)
What about the blocking google bot by their IPs, also combined with user-agent wouldn't that stop the crawlers
Google crawlers IPs https://www.lifewire.com/what-is-the-ip-address-of-google-81...
That will stop the crawlers but you could still show up in the search results, because of other web pages. From GP:
> If other pages point to your page with descriptive text, Google could still index the URL without visiting the page
“ For those of you who relied on the noindex indexing directive in the robots.txt file, which controls crawling, there are a number of alternative options:”
The first option is the meta tag. It does mention an alternative directive for robots.txt, however.