News for hackers. Stuff that matters.

A comment on Addressing Recent Claims of “Manipulated” Blog Posts in the Wayback Machine. Parent comment

Re:Addressing Recent Claims of “Manipulated” Blog Posts in t... by klez on Wednesday April 25, 2018 @04:14AM (#16919482)
They seem to have had this problem in the past and decided to skirt around it by ignoring robots.txt a year ago[0]. Does anybody know what happened to revert this decision?

[0]https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

[ Parent | on HN ]
Re:Addressing Recent Claims of “Manipulated” Blog Posts in t... by icebraining on Wednesday April 25, 2018 @04:49AM (#16919653)
As per that post, they only ignored robots.txt for .gov and .mil sites.
[ Parent | on HN ]
Re:Addressing Recent Claims of “Manipulated” Blog Posts in t... by aero-002 on Wednesday April 25, 2018 @05:10AM (#16919765)
IA disallow in robots.txt will still block archive.org, the blog post was about ignoring parts that were meant for search engines.
[ Parent | on HN ]
Re:Addressing Recent Claims of “Manipulated” Blog Posts in t... by klez on Wednesday April 25, 2018 @04:59AM (#16919701)
Yes, but it also says

> We are now looking to do this more broadly.

That's the part I'm asking about.

[ Parent | on HN ]
Re:Addressing Recent Claims of “Manipulated” Blog Posts in t... by icebraining on Wednesday April 25, 2018 @05:02AM (#16919721)
Right, but it doesn't mean they reverted it, they are probably still looking into it.
[ Parent | on HN ]