News for hackers. Stuff that matters.

A comment on Wikipedia's robots.txt. Parent comment

Re:Wikipedia's robots.txt by CydeWeys on Tuesday July 16, 2019 @04:10PM (#20453447)
Google, Bing, and other large search engines honor it. That alone is the point of most of the entries in this file.
[ Parent | on HN ]

There are no comments on this comment yet.