News for hackers. Stuff that matters.

A comment on Robots.txt for the NYT has a specific exclusion for an 1996 news article. Parent comment

Re:Robots.txt for the NYT has a specific exclusion for an 19... by taway232345 on Monday March 29, 2021 @01:33AM (#26618013)
That is one lucky professor -- how did she get granted an exception? There are tons of unlucky people who suffer as a result of bad/shoddy/vague reporting, how do they also get a carve-out?
[ Parent | on HN ]
Re:Robots.txt for the NYT has a specific exclusion for an 19... by gostsamo on Monday March 29, 2021 @02:00AM (#26618139)
In my case, nobody was actually malicious. It was a stupid situation caused by the stupid actions of both sides from what I've been able to discover. Nobody meant to cause harm but to tell their own story. I tried to find a compromise that would work for everyone and it worked this time.
[ Parent | on HN ]
Re:Robots.txt for the NYT has a specific exclusion for an 19... by MattGaiser on Monday March 29, 2021 @02:01AM (#26618140)
Like many people who get exceptions. They asked.
[ Parent | on HN ]