News for hackers. Stuff that matters.

A comment on Robots.txt for the NYT has a specific exclusion for an 1996 news article. Parent comment

Re:Robots.txt for the NYT has a specific exclusion for an 19... by remux on Monday March 29, 2021 @05:30AM (#26619218)
This was changed in 2019: https://www.robots-viewer.com/robots/checksum/64dc55e4da9cc8...

robots.txt version from 2017: https://www.robots-viewer.com/robots/checksum/8029662cfb040c...

[ Parent | on HN ]
Re:Robots.txt for the NYT has a specific exclusion for an 19... by makomk on Monday March 29, 2021 @07:51AM (#26620265)
Probably because that's around the time he seemingly really pissed off one or more former employees who went digging for more information on him: https://www.glassdoor.com/Reviews/Employee-Review-Romio-RVW2...
[ Parent | on HN ]