News for hackers. Stuff that matters.
Re:Wikipedia's robots.txt by wybiral on Tuesday July 16, 2019 @06:07PM (#20454327)
GitHub has an interesting entry in theirs [0]. The only user that's in the Disallow section is "/ekansa".

He explained it a bit on a Twitter thread [1] where I brought it up a while back and it sounds like they didn't like the fact that he was using GitHub to host XML files for another service because of the traffic from crawlers it created.

[0] https://github.com/robots.txt

[1] https://twitter.com/ekansa/status/1137052076062650368

Re:Wikipedia's robots.txt by philpem on Wednesday July 17, 2019 @04:02AM (#20457447)
User-agent: *

Allow: /humans.txt

Disallow: /

Nice!