Re:Wikipedia's robots.txt
by wybiral on Tuesday July 16, 2019 @06:07PM
(#20454327)
GitHub has an interesting entry in theirs [0]. The only user that's in the Disallow section is "/ekansa".
He explained it a bit on a Twitter thread [1] where I brought it up a while back and it sounds like they didn't like the fact that he was using GitHub to host XML files for another service because of the traffic from crawlers it created.
He explained it a bit on a Twitter thread [1] where I brought it up a while back and it sounds like they didn't like the fact that he was using GitHub to host XML files for another service because of the traffic from crawlers it created.
[0] https://github.com/robots.txt
[1] https://twitter.com/ekansa/status/1137052076062650368