News for hackers. Stuff that matters.
Re:The Internet Archive is waging war on misinformation by ineedasername on Wednesday September 18, 2019 @03:19PM (#21008936)
I like that the internet archive exists, but I don't like how they implement their opt-out policy. Sites can opt-out at any time, which I suppose is the correct thing to allow, but once they opt out then everything the Archive has gathered up to that point also goes away, which seems wrong.
Re:The Internet Archive is waging war on misinformation by Rebelgecko on Wednesday September 18, 2019 @03:36PM (#21009139)
I got frustrated with this one time when I tried to look up a defunct website in the Wayback Machine. The website was gone, but had been replaced by a domain squatter with a restrictive robots.txt. Because of the new roboots.txt, all of the old site was retroactively made unavailable. As time passes and more websites change hands, it would be a nice compromise if the default behavior was only to make retroactive deletions that go back to the last change of ownership in WHOIS data.
Re:The Internet Archive is waging war on misinformation by dredmorbius on Wednesday September 18, 2019 @05:43PM (#21010568)
IA are specifically refusing to follow robots.txt directives largely on account of just this behaviour.

https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

https://archive.org/post/184024/robotstxt-policy-is-a-failur...

Re:The Internet Archive is waging war on misinformation by Rebelgecko on Wednesday September 18, 2019 @06:23PM (#21010998)
That's great news. Is there any update to the 2017 blog post? Is the new policy still only for .mil and .gov sites?
Re:The Internet Archive is waging war on misinformation by dredmorbius on Wednesday September 18, 2019 @06:39PM (#21011186)
I don't know, though you could research via their blogsite.
Re:The Internet Archive is waging war on misinformation by foota on Wednesday September 18, 2019 @04:43PM (#21009951)
Would be cool if this could be tied to SSL keys. Being able to serve requests with the same SSL key could be used as verification. (obviously some edge cases like people that are behind some kind of common SSL cert like a subdomain with shared hosting)
Re:The Internet Archive is waging war on misinformation by loa_in_ on Wednesday September 18, 2019 @07:54PM (#21011809)
That's impossible due to the way assymetric encryption works. It would help to indicate to the user, and maybe allow searching/grouping by certificate hash to transcend domain names
Re:The Internet Archive is waging war on misinformation by foota on Thursday September 19, 2019 @02:33PM (#21019120)
Isn't that what cert pinning does though?
Re:The Internet Archive is waging war on misinformation by chousuke on Thursday September 19, 2019 @03:50PM (#21019963)
I think it would make more sense to tie the behaviour to the domain's whois record. If that changes significantly, chances are the domain has changed owners.
Re:The Internet Archive is waging war on misinformation by foota on Thursday September 19, 2019 @06:23PM (#21021354)
This is true, but the whois isn't really a highly official source.
Re:The Internet Archive is waging war on misinformation by dredmorbius on Friday September 20, 2019 @06:29AM (#21025090)
Actually, it is. It's the TLD registrar's record.

The usefulness of WHOIS has been declining precipitously, largely as useful information has been horribly misused and abused.

Re:The Internet Archive is waging war on misinformation by mmagin on Wednesday September 18, 2019 @04:35PM (#21009852)
To be fair, they're operating in a situation (budget vs. volume of material to collect and share with the world) where it's necessary to minimize the human overhead of this mechanism AND to attempt to reduce their legal exposure -- sure they can get legal help from some of the very best, but mostly out of people's goodwill, they don't have the budget to be defending themselves constantly.

(Disclaimer: former IA employee, but I do not in any way speak for IA.)

Re:The Internet Archive is waging war on misinformation by mjevans on Wednesday September 18, 2019 @04:24PM (#21009716)
I believe that policy to be morally wrong. If a site WAS accessible then that old version should continue to be until an active legal tool is used to request the offline archiving rather than free access as it was at the time of observation.
Re:The Internet Archive is waging war on misinformation by blackflame7000 on Wednesday September 18, 2019 @04:50PM (#21010038)
Yea but I can't just copy a website and start hosting my own version for numerous copyright reasons even if it is just to archive. I can't for example torrent a bunch of movies and store them for the purpose of "archiving"
Re:The Internet Archive is waging war on misinformation by ticmasta on Wednesday September 18, 2019 @04:55PM (#21010088)
Actually, a library is a specific legal context that does allow it to do many things similar to what you're describing. It can't steal movies for archiving, but it certainly can purchase and share legally procured copies. Digital sharing is an admittedly more complex minefield...
Re:The Internet Archive is waging war on misinformation by aidenn0 on Wednesday September 18, 2019 @05:14PM (#21010293)
While libraries do have special protections, sharing legally procured physical copies is more broadly protected due to the first-sale doctrine.

However, libraries do have an exemption to the ban on renting music recordings (after the broad availability of audio cassette recorders, congress made renting music illegal, but libraries and schools can still loan them out).

Re:The Internet Archive is waging war on misinformation by pbhjpbhj on Wednesday September 18, 2019 @05:11PM (#21010268)
In USA. I assume that's where you're talking about.

In UK certain libraries can archive (British Library) following a recent change in the law, IIRC. People wanting to access the archive may only do so by physically visiting the library.

AFAICT archive.org are flouting copyright law around the World already, they should continue to be cautious about assuming rights IMO.

Re:The Internet Archive is waging war on misinformation by kevin_thibedeau on Wednesday September 18, 2019 @05:33PM (#21010461)
Archive.org is an accredited library.
Re:The Internet Archive is waging war on misinformation by pbhjpbhj on Wednesday September 18, 2019 @06:09PM (#21010833)
Yep, that works for USA law AFAIR, but USA Fair Use is exceptionally liberal compared to other copyright regimes.
Re:The Internet Archive is waging war on misinformation by dTal on Thursday September 19, 2019 @07:16AM (#21015159)
That's exactly what they do. They are treating "robots.txt" as an "active legal tool". You could argue that they should make it harder, but I doubt they want to find themselves in court over every little thing. Better to make it easy to block to avoid liability, and also easy to allow that block to lapse - if they forced you to get a court order to hide information, then the information would be gone forever.
Re:The Internet Archive is waging war on misinformation by mjevans on Thursday September 19, 2019 @04:02PM (#21020115)
An active legal tool might be as little as requiring an official claim of copyright and a related Cease and Desist to be mailed physically to them.

That would prevent the issue of domain squatters with robots.txt files in most cases, and offer the public a means of challenging such ownership requests (and freeing up the content on archive.org) if they desire.

Re:The Internet Archive is waging war on misinformation by maxerickson on Thursday September 19, 2019 @09:33AM (#21015978)
Well get to it then.
Re:The Internet Archive is waging war on misinformation by ent101 on Wednesday September 18, 2019 @04:05PM (#21009485)
Just to be clear, the data doesn't seem to be deleted -- it only becomes inaccessible once the site opts out; all the data can come back if the site changes its mind.
Re:The Internet Archive is waging war on misinformation by pishpash on Wednesday September 18, 2019 @06:00PM (#21010732)
Does it also continue to archive while the 'do not archive' flag is on?
Re:The Internet Archive is waging war on misinformation by ent101 on Wednesday September 18, 2019 @10:40PM (#21012739)
I'm not sure, but I do think they follow the standard guidelines, meaning they won't archive if the flag is on. This can be verified by going through the Heritrix crawler's source code.
Re:The Internet Archive is waging war on misinformation by toomuchtodo on Wednesday September 18, 2019 @03:23PM (#21008995)
It doesn’t go away, it’s just not public. This is to comply with copyright law. It’s still safely stored on disk.
Re:The Internet Archive is waging war on misinformation by bryanrasmussen on Wednesday September 18, 2019 @04:19PM (#21009667)
Not sure about that copyright argument holding water, especially across multiple jurisdictions - I would think some argument could be made that the data was gathered with a robots.txt that allowed it and as such it can be made available.

Actually as I've noted before other archive initiatives often ignore robots.txt, for example the Danish National Archive ignores it and is legally allowed to for gathering all relevant Danish material.

Re:The Internet Archive is waging war on misinformation by ineedasername on Wednesday September 18, 2019 @03:34PM (#21009117)
Okay, if they have no choice in it then I understand. But as for keeping a copy on disk that practically no one can ever see, to me that strikes seems like a distinction without a difference.
Re:The Internet Archive is waging war on misinformation by diggan on Wednesday September 18, 2019 @03:38PM (#21009159)
I'm pretty sure you can visit the building that Internet Archive operates from (https://archive.org/about/contact.php) and freely surf/look at all the content, even the content that is not available from the web version.
Re:The Internet Archive is waging war on misinformation by raxxorrax on Thursday September 19, 2019 @01:59AM (#21013690)
I see no advantage in restricting access. I just believe this to be a mechanism to hide uncomfortable content, which I believe has no benefit as long as no personal information is revealed.
Re:The Internet Archive is waging war on misinformation by emiliobumachar on Wednesday September 18, 2019 @03:48PM (#21009264)
It's not a binary fact of having a choice or not. A less restrictive policy would bring a load of frivolous copyright and/or trademark encroachment. Fighting it would be a worthy fight, but would divert resources away from their core mission of archiving, which is hard enough on its own.

AFAIK they're thinking of the historical record for future generations, and present access is a sacrificeable side benefit.

Re:The Internet Archive is waging war on misinformation by toomuchtodo on Wednesday September 18, 2019 @03:44PM (#21009220)
Better to save content for a future with more sane copyright law than to not save it at all.
Re:The Internet Archive is waging war on misinformation by dTal on Thursday September 19, 2019 @07:17AM (#21015167)
The block requires active maintenance in the form of robots.txt. If the domain owner allows robots.txt to lapse, the content is automatically republished.
Re:The Internet Archive is waging war on misinformation by pgeorgi on Wednesday September 18, 2019 @04:44PM (#21009963)
"No one" in the next ~70-90 years.
Re:The Internet Archive is waging war on misinformation by gizmo686 on Thursday September 19, 2019 @10:41AM (#21016618)
One of the services they are providing is to future historians.
Re:The Internet Archive is waging war on misinformation by jMyles on Wednesday September 18, 2019 @03:52PM (#21009325)
> This is to comply with copyright law.

If that's true, it will be a better resource for humanity if it charts a way to be hosted in a jurisdiction where copyright law doesn't require this kind of censorship.

Re:The Internet Archive is waging war on misinformation by toomuchtodo on Wednesday September 18, 2019 @06:08PM (#21010818)
Re:The Internet Archive is waging war on misinformation by dddddaviddddd on Wednesday September 18, 2019 @03:53PM (#21009341)
Ready for public release in roughly 100 years then.
Re:The Internet Archive is waging war on misinformation by pbhjpbhj on Wednesday September 18, 2019 @05:24PM (#21010375)
It doesn't make then compliant [with UK law, at least], it just makes them appear to be. Copyright covers copying, the copying has already been done, and the copies are being retained: meaning any infringement is ongoing. They're just hiding their probable infringement.

I'm more than happy for them to do that btw, but I do have strong reservations about letting anyone "archive" a site and offer it to the public. This is an area that's needs legislative development IMO.

Re:The Internet Archive is waging war on misinformation by toomuchtodo on Wednesday September 18, 2019 @05:37PM (#21010501)
The Internet Archive has no physical presence in the UK to my knowledge.
Re:The Internet Archive is waging war on misinformation by pbhjpbhj on Wednesday September 18, 2019 @06:08PM (#21010810)
They copy websites from UK servers. They extend their presence to UK jurisdiction in order to do that.
Re:The Internet Archive is waging war on misinformation by colejohnson66 on Wednesday September 18, 2019 @08:05PM (#21011885)
So when I copy a UK website from to my browser cache, I’m subject to UK jurisdiction? That makes no sense.
Re:The Internet Archive is waging war on misinformation by dTal on Thursday September 19, 2019 @07:23AM (#21015193)
The archive needs to provide some extra-legal mechanism to allow people to hide data - otherwise they'll be in court all the time. The advantage of this system is that it runs both ways - sites can also neglect to opt out, having previously done so, which immediately republishes all the data.