News for hackers. Stuff that matters.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by kalleboo on Wednesday January 25, 2017 @09:23PM (#13488207)
The internet archive has made this their mission.

Archive Team keeps a constant eye out for major content sites that are at risk of being shut down and proactively mirroring their content. http://www.archiveteam.org/index.php?title=Main_Page

On EPA data https://twitter.com/textfiles/status/824110893034311680

They don't just archive online content either. They spend many resources scanning magazines, ingesting (and cataloging) CDs and floppies before they all rot away.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by astrodust on Wednesday January 25, 2017 @09:31PM (#13488236)
As much as it's probably a logistical nightmare, they basically need to back up everything now. This is triage time because everything is at risk of being shut down.

You know when the Internet Archive itself is making a contingency plan to move out of the US that things are truly fucked.

The most difficult thing here is any prirotization they end up doing must be kept absolutely secret or you're going to be providing a hit-list to the administration should they ever want one.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by barney54 on Wednesday January 25, 2017 @10:54PM (#13488573)
Where is the evidence that any of the data sites are at risk of being shutdown?
Re:Public appeal to backup Climate Monitoring and Diagnostic... by lazerwalker on Wednesday January 25, 2017 @11:14PM (#13488644)
I don't think there's direct concrete evidence, but that the Internet Archive themselves just ran a crowdfunding campaign to open a redundant data center in Canada suggests it is something they are worried about.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by ianai on Wednesday January 25, 2017 @11:56PM (#13488869)
Whole departments of the US government being told to hush up, including in peer reviewed journals, is direct concrete evidence.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by closeparen on Wednesday January 25, 2017 @11:12PM (#13488636)
Won't they delete it all when epa.gov puts up a robots.txt?
Re:Public appeal to backup Climate Monitoring and Diagnostic... by af16090 on Wednesday January 25, 2017 @11:37PM (#13488768)
My understanding is that they will disallow access to a webpage if a robots.txt goes up but they will keep the copy of the webpage in case the robots.txt is ever changed[1]. So a future administration could change the robots.txt and the webpages would be accessible again.

[1]: Seems like that happened here: https://en.wikipedia.org/wiki/Wayback_Machine#Netbula_LLC_v.....

Re:Public appeal to backup Climate Monitoring and Diagnostic... by hunter2_ on Wednesday January 25, 2017 @11:59PM (#13488883)
I know robots.txt is all about respect in the first place, but is it really expected that obtained-via-robot mirrors will be made inaccessible to humans during periods of time when the upstream prohibits robotic fetching following that original fetch? It seems much more reasonable that the mirror would merely prohibit robots, not humans, from fetching its copy during periods of time when the upstream prohibits robots.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by CM30 on Thursday January 26, 2017 @06:41AM (#13490554)
I suspect the original thought was that a website owner might later decide they don't want their site publically viewable in the Internet Archive, and would sue/complain if it was still visible there. Like say, if they accidentally allowed robots to archive a 'hidden' directory.

Problem is, those use cases are greatly outweighed by ones where either:

1. The site has changed ownership to a different (legitimate) company or organisation, and they don't want their new site archived.

2. A domain squatter/seller has bought the domain, blocked all bots to stop the holding page being indexed in Google and accidentally blocked the archive in the process.

3. A technician or developer has accidentally blocked the archive/all robots due to a personal mistake/copying code from the internet.

4. The site owner doesn't know the archive has a bot, and has blocked all non Google/Bing bots to 'reduce strain' on the server. That last one is depressingly common:

http://webmasters.stackexchange.com/questions/75993/only-all...

Re:Public appeal to backup Climate Monitoring and Diagnostic... by CM30 on Thursday January 26, 2017 @06:36AM (#13490538)
Fortunately, they ignore robots.txt for .gov and .mil sites:

https://twitter.com/internetarchive/status/82406378479265792...

So in this case, the Internet Archive should work fine.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by astrodust on Thursday January 26, 2017 @12:42PM (#13493406)
That's a really sensible policy and an especially forward-thinking one.