News for hackers. Stuff that matters.
Re:Fastly Outage by oefrha on Tuesday June 08, 2021 @08:14AM (#27434092)
That's not going to work.

  $ curl https://old.reddit.com/robots.txt
  User-Agent: *
  Disallow: /
Also, even if search engines are allowed, old.reddit.com pages are not canonical (<link rel="canonical"> points to the www.reddit.com version, which is actually reasonable behavior), so pages there would not be crawled as often or at all.

There are no comments on this comment yet.