Re:How Google determined our right to be forgotten
by benologist on Wednesday February 18, 2015 @04:04PM
(#9070920)
For all intents and purposes "indexing" and "crawling" is "appearing in search results", and they conflate these terms themselves in their robots.txt page:
> "You only need a robots.txt file if your site includes content that you don't want Google or other search engines to index."
They do say noindex overrides 3rd party links so that's something, but it's a moot point either way because this was the approach for the last decade. Having to contact each individual person in charge of each individual URL isn't even effective when the MPAA throws money+DMCA at the task, individuals asking nicely never had a chance.
> "You only need a robots.txt file if your site includes content that you don't want Google or other search engines to index."
https://support.google.com/webmasters/answer/6062608?hl=en
They do say noindex overrides 3rd party links so that's something, but it's a moot point either way because this was the approach for the last decade. Having to contact each individual person in charge of each individual URL isn't even effective when the MPAA throws money+DMCA at the task, individuals asking nicely never had a chance.