Re:The Internet Archive is waging war on misinformation
by Rebelgecko on Wednesday September 18, 2019 @03:36PM
(#21009139)
I got frustrated with this one time when I tried to look up a defunct website in the Wayback Machine. The website was gone, but had been replaced by a domain squatter with a restrictive robots.txt. Because of the new roboots.txt, all of the old site was retroactively made unavailable. As time passes and more websites change hands, it would be a nice compromise if the default behavior was only to make retroactive deletions that go back to the last change of ownership in WHOIS data.
[ Parent
| on HN ]
Re:The Internet Archive is waging war on misinformation
by dredmorbius on Wednesday September 18, 2019 @05:43PM
(#21010568)
IA are specifically refusing to follow robots.txt directives largely on account of just this behaviour.
https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
https://archive.org/post/184024/robotstxt-policy-is-a-failur...
Re:The Internet Archive is waging war on misinformation
by Rebelgecko on Wednesday September 18, 2019 @06:23PM
(#21010998)
That's great news. Is there any update to the 2017 blog post? Is the new policy still only for .mil and .gov sites?
Re:The Internet Archive is waging war on misinformation
by dredmorbius on Wednesday September 18, 2019 @06:39PM
(#21011186)
I don't know, though you could research via their blogsite.
Re:The Internet Archive is waging war on misinformation
by foota on Wednesday September 18, 2019 @04:43PM
(#21009951)
Would be cool if this could be tied to SSL keys. Being able to serve requests with the same SSL key could be used as verification. (obviously some edge cases like people that are behind some kind of common SSL cert like a subdomain with shared hosting)
Re:The Internet Archive is waging war on misinformation
by loa_in_ on Wednesday September 18, 2019 @07:54PM
(#21011809)
That's impossible due to the way assymetric encryption works. It would help to indicate to the user, and maybe allow searching/grouping by certificate hash to transcend domain names
Re:The Internet Archive is waging war on misinformation
by foota on Thursday September 19, 2019 @02:33PM
(#21019120)
Isn't that what cert pinning does though?
Re:The Internet Archive is waging war on misinformation
by chousuke on Thursday September 19, 2019 @03:50PM
(#21019963)
I think it would make more sense to tie the behaviour to the domain's whois record. If that changes significantly, chances are the domain has changed owners.
Re:The Internet Archive is waging war on misinformation
by foota on Thursday September 19, 2019 @06:23PM
(#21021354)
This is true, but the whois isn't really a highly official source.
Re:The Internet Archive is waging war on misinformation
by dredmorbius on Friday September 20, 2019 @06:29AM
(#21025090)
Actually, it is. It's the TLD registrar's record.
The usefulness of WHOIS has been declining precipitously, largely as useful information has been horribly misused and abused.