News for hackers. Stuff that matters.
Re:Wikipedia's robots.txt by mzmcbride on Tuesday July 16, 2019 @06:16PM (#20454378)
This URL was returning "Not Found" for me as well, but with an HTTP 501 status instead of the traditional HTTP 404. After investigating, the issue seems to be cookie-related:

  $ curl -sIXGET 'https://www.reddit.com/robots.txt' -H 'cookie: rseor3=true;' | head -1
  HTTP/2 501
Compare with:

  $ curl -sIXGET 'https://www.reddit.com/robots.txt' | head -1
  HTTP/2 200
Re:Wikipedia's robots.txt by kauegimenes on Tuesday July 16, 2019 @06:33PM (#20454495)
Thanks, works from incognito mode in browsers also then :)