News for hackers. Stuff that matters.

A comment on Addressing Recent Claims of “Manipulated” Blog Posts in the Wayback Machine. Parent comment

Re:Addressing Recent Claims of “Manipulated” Blog Posts in t... by htgb on Wednesday April 25, 2018 @03:30AM (#16919271)
> (Because the Wayback Machine starts from scratch each time?)

I don't know what "start from scratch" would mean – the point is that each site is sampled many times throughout history. That said, it is very odd that a current change in robots.txt would prevent looking at old samples. And that's indeed what it looks like [1]:

> Page cannot be displayed due to robots.txt.

[1] https://web.archive.org/web/*/blog.reidreport.com

[ Parent | on HN ]
Re:Addressing Recent Claims of “Manipulated” Blog Posts in t... by pbhjpbhj on Wednesday April 25, 2018 @06:04AM (#16920070)
I'd imagine they're in a dodgy copyright situation and so guard against it by being conservative wrt robots.txt.

The robots.txt shows a positive assertion that parts of a site should be excluded from being used by automated systems.

In most cases I imagine WBM does not have permission of the owner to keep a duplicate of the site, it's certainly tortuous in UK law.

Sites that don't change their robots.txt are probably highly correlated with sites that don't sue for the infringement.

[ Parent | on HN ]