News for hackers. Stuff that matters.

A comment on Wikipedia's robots.txt. Parent comment

Re:Wikipedia's robots.txt by dTal on Tuesday July 16, 2019 @06:39PM (#20454553)
A whitelist of FAANG crawlers would cement their hegemony - a blacklist of known-badly-behaved crawlers doesn't.
[ Parent | on HN ]

There are no comments on this comment yet.