Wikipedia's robots.txt

Re:Wikipedia's robots.txt by mistrial9 on Tuesday July 16, 2019 @05:38PM (#20454135)

bulk filtering like that only cements FAANG hegemony; agree its a problem, do not agree this is the solution

[ Parent | on HN ]

Re:Wikipedia's robots.txt by dTal on Tuesday July 16, 2019 @06:39PM (#20454553)

A whitelist of FAANG crawlers would cement their hegemony - a blacklist of known-badly-behaved crawlers doesn't.

[ Parent | on HN ]

Re:Wikipedia's robots.txt by nkozyra on Tuesday July 16, 2019 @08:20PM (#20455216)

I don't filter out anyone but bad actors. If you abide by robots.txt you're free to scrape my sites

[ Parent | on HN ]