Robots.txt for the NYT has a specific exclusion for an 1996 news article

Re:Robots.txt for the NYT has a specific exclusion for an 19... by rurban on Monday March 29, 2021 @06:45AM (#26619793)

This was one of the very rare cases where even wikileaks took down one article in the global intelligence files, after the state dpmt complained. A very high profile case. In that specific country everybody knew, what the diplomate of the other very specific country did.

[ Parent | on HN ]

Re:Robots.txt for the NYT has a specific exclusion for an 19... by bigbillheck on Monday March 29, 2021 @10:38AM (#26622315)

Is it related to this particular incident? I'm not finding anything immediate.

[ Parent | on HN ]

Re:Robots.txt for the NYT has a specific exclusion for an 19... by rurban on Monday March 29, 2021 @01:30PM (#26625274)

You wont find anything, as it was taken down a few days after publication. The NY Times article was taken down later, but had not much juicy info.

[ Parent | on HN ]