News for hackers. Stuff that matters.
Re:Wikipedia's robots.txt by jedberg on Tuesday July 16, 2019 @04:32PM (#20453597)
This seems unnecessarily complicated and antagonistic. Mostly just to publicly shame a bunch of people.

At Reddit we originally blocked a couple of crawlers but then realized how pointless that was. The entire robots file[0] is basically now just for google. All of the restrictions are enforced on the server side because there were so many bad bots it didn’t matter if we listed them or not.

[0] https://www.reddit.com/robots.txt

Re:Wikipedia's robots.txt by wybiral on Tuesday July 16, 2019 @06:07PM (#20454327)
GitHub has an interesting entry in theirs [0]. The only user that's in the Disallow section is "/ekansa".

He explained it a bit on a Twitter thread [1] where I brought it up a while back and it sounds like they didn't like the fact that he was using GitHub to host XML files for another service because of the traffic from crawlers it created.

[0] https://github.com/robots.txt

[1] https://twitter.com/ekansa/status/1137052076062650368

Re:Wikipedia's robots.txt by philpem on Wednesday July 17, 2019 @04:02AM (#20457447)
User-agent: *

Allow: /humans.txt

Disallow: /

Nice!

Re:Wikipedia's robots.txt by notafrog on Tuesday July 16, 2019 @04:44PM (#20453699)
I was totally expecting /my_shiny_metal_ass to have some kind of easter egg.
Re:Wikipedia's robots.txt by jedberg on Tuesday July 16, 2019 @07:31PM (#20454908)
We used to have one a long time ago when we fronted with nginx and I could just throw a static file up there, but once we switched to everything being dynamic, it wasn't worth the effort to code up a route. :)
Re:Wikipedia's robots.txt by thinkloop on Tuesday July 16, 2019 @04:41PM (#20453670)
> Mostly just to publicly shame a bunch of people.

For sure, but that's great, shame can be very effective.

Re:Wikipedia's robots.txt by kauegimenes on Tuesday July 16, 2019 @04:57PM (#20453821)
https://www.reddit.com/robots.txt replies Not Found over here
Re:Wikipedia's robots.txt by mzmcbride on Tuesday July 16, 2019 @06:16PM (#20454378)
This URL was returning "Not Found" for me as well, but with an HTTP 501 status instead of the traditional HTTP 404. After investigating, the issue seems to be cookie-related:

  $ curl -sIXGET 'https://www.reddit.com/robots.txt' -H 'cookie: rseor3=true;' | head -1
  HTTP/2 501
Compare with:

  $ curl -sIXGET 'https://www.reddit.com/robots.txt' | head -1
  HTTP/2 200
Re:Wikipedia's robots.txt by kauegimenes on Tuesday July 16, 2019 @06:33PM (#20454495)
Thanks, works from incognito mode in browsers also then :)