News for hackers. Stuff that matters.

A comment on Robots.txt for the NYT has a specific exclusion for an 1996 news article. Parent comment

Re:Robots.txt for the NYT has a specific exclusion for an 19... by rafaelm on Monday March 29, 2021 @12:15AM (#26617648)
That's right, but the url exclusion tool is only temporary, so the only way to do it correctly is with the noindex tag.
[ Parent | on HN ]

There are no comments on this comment yet.