News for hackers. Stuff that matters.
Re:Robots.txt for the NYT has a specific exclusion for an 19... by snowwrestler on Monday March 29, 2021 @12:06AM (#26617617)
Just for future reference: adding a URL to robots.txt will not necessarily exclude that web page from Google, especially if it has already been indexed.

To reliably exclude a URL from indexing, you have to serve a “no index” instruction with that URL, either in a meta tag or an HTTP header. And for this instruction to be read, the robot has to visit that page! So disallowing the URL in robots.txt can actually be counterproductive to de-indexing it.

Google also offers a tool specifically for removing URLs from their index in Search Console.

Re:Robots.txt for the NYT has a specific exclusion for an 19... by rafaelm on Monday March 29, 2021 @12:15AM (#26617648)
That's right, but the url exclusion tool is only temporary, so the only way to do it correctly is with the noindex tag.