GitHub has an interesting entry in theirs [0]. The only user that's in the Disallow section is "/ekansa".
He explained it a bit on a Twitter thread [1] where I brought it up a while back and it sounds like they didn't like the fact that he was using GitHub to host XML files for another service because of the traffic from crawlers it created.
User-agent: *
Allow: /humans.txt
Disallow: /
Nice!
I was totally expecting /my_shiny_metal_ass to have some kind of easter egg.
We used to have one a long time ago when we fronted with nginx and I could just throw a static file up there, but once we switched to everything being dynamic, it wasn't worth the effort to code up a route. :)
> Mostly just to publicly shame a bunch of people.
For sure, but that's great, shame can be very effective.
Re:Wikipedia's robots.txt
by kauegimenes on Tuesday July 16, 2019 @04:57PM
(#20453821)
https://www.reddit.com/robots.txt replies Not Found over here
This URL was returning "Not Found" for me as well, but with an HTTP 501 status instead of the traditional HTTP 404. After investigating, the issue seems to be cookie-related:
$ curl -sIXGET 'https://www.reddit.com/robots.txt' -H 'cookie: rseor3=true;' | head -1
HTTP/2 501
Compare with: $ curl -sIXGET 'https://www.reddit.com/robots.txt' | head -1
HTTP/2 200
Re:Wikipedia's robots.txt
by kauegimenes on Tuesday July 16, 2019 @06:33PM
(#20454495)
Thanks, works from incognito mode in browsers also then :)
At Reddit we originally blocked a couple of crawlers but then realized how pointless that was. The entire robots file[0] is basically now just for google. All of the restrictions are enforced on the server side because there were so many bad bots it didn’t matter if we listed them or not.
[0] https://www.reddit.com/robots.txt