Re:Robots.txt for the NYT has a specific exclusion for an 19...
by OJFord on Monday March 29, 2021 @08:24AM
(#26620513)
Accomplishes the same, but it's a different mechanism.
RTBF does not use robots.txt (and then require that it be respected) on the site hosting the thing to be forgotten, it's an exclusion on the search engine side.
RTBF does not use robots.txt (and then require that it be respected) on the site hosting the thing to be forgotten, it's an exclusion on the search engine side.