News for hackers. Stuff that matters.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by astrodust on Wednesday January 25, 2017 @09:20PM (#13488192)
We should be mirroring fucking everything.

Where's a good place to put data like this? Is there even a place where people can start to inventory what's at risk?

The biggest problem is not knowing what's out there, what records need to be protected, as there's so many people doing important work.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by kalleboo on Wednesday January 25, 2017 @09:23PM (#13488207)
The internet archive has made this their mission.

Archive Team keeps a constant eye out for major content sites that are at risk of being shut down and proactively mirroring their content. http://www.archiveteam.org/index.php?title=Main_Page

On EPA data https://twitter.com/textfiles/status/824110893034311680

They don't just archive online content either. They spend many resources scanning magazines, ingesting (and cataloging) CDs and floppies before they all rot away.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by astrodust on Wednesday January 25, 2017 @09:31PM (#13488236)
As much as it's probably a logistical nightmare, they basically need to back up everything now. This is triage time because everything is at risk of being shut down.

You know when the Internet Archive itself is making a contingency plan to move out of the US that things are truly fucked.

The most difficult thing here is any prirotization they end up doing must be kept absolutely secret or you're going to be providing a hit-list to the administration should they ever want one.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by barney54 on Wednesday January 25, 2017 @10:54PM (#13488573)
Where is the evidence that any of the data sites are at risk of being shutdown?
Re:Public appeal to backup Climate Monitoring and Diagnostic... by lazerwalker on Wednesday January 25, 2017 @11:14PM (#13488644)
I don't think there's direct concrete evidence, but that the Internet Archive themselves just ran a crowdfunding campaign to open a redundant data center in Canada suggests it is something they are worried about.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by ianai on Wednesday January 25, 2017 @11:56PM (#13488869)
Whole departments of the US government being told to hush up, including in peer reviewed journals, is direct concrete evidence.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by closeparen on Wednesday January 25, 2017 @11:12PM (#13488636)
Won't they delete it all when epa.gov puts up a robots.txt?
Re:Public appeal to backup Climate Monitoring and Diagnostic... by af16090 on Wednesday January 25, 2017 @11:37PM (#13488768)
My understanding is that they will disallow access to a webpage if a robots.txt goes up but they will keep the copy of the webpage in case the robots.txt is ever changed[1]. So a future administration could change the robots.txt and the webpages would be accessible again.

[1]: Seems like that happened here: https://en.wikipedia.org/wiki/Wayback_Machine#Netbula_LLC_v.....

Re:Public appeal to backup Climate Monitoring and Diagnostic... by hunter2_ on Wednesday January 25, 2017 @11:59PM (#13488883)
I know robots.txt is all about respect in the first place, but is it really expected that obtained-via-robot mirrors will be made inaccessible to humans during periods of time when the upstream prohibits robotic fetching following that original fetch? It seems much more reasonable that the mirror would merely prohibit robots, not humans, from fetching its copy during periods of time when the upstream prohibits robots.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by CM30 on Thursday January 26, 2017 @06:41AM (#13490554)
I suspect the original thought was that a website owner might later decide they don't want their site publically viewable in the Internet Archive, and would sue/complain if it was still visible there. Like say, if they accidentally allowed robots to archive a 'hidden' directory.

Problem is, those use cases are greatly outweighed by ones where either:

1. The site has changed ownership to a different (legitimate) company or organisation, and they don't want their new site archived.

2. A domain squatter/seller has bought the domain, blocked all bots to stop the holding page being indexed in Google and accidentally blocked the archive in the process.

3. A technician or developer has accidentally blocked the archive/all robots due to a personal mistake/copying code from the internet.

4. The site owner doesn't know the archive has a bot, and has blocked all non Google/Bing bots to 'reduce strain' on the server. That last one is depressingly common:

http://webmasters.stackexchange.com/questions/75993/only-all...

Re:Public appeal to backup Climate Monitoring and Diagnostic... by CM30 on Thursday January 26, 2017 @06:36AM (#13490538)
Fortunately, they ignore robots.txt for .gov and .mil sites:

https://twitter.com/internetarchive/status/82406378479265792...

So in this case, the Internet Archive should work fine.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by astrodust on Thursday January 26, 2017 @12:42PM (#13493406)
That's a really sensible policy and an especially forward-thinking one.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by sandworm101 on Wednesday January 25, 2017 @09:40PM (#13488275)
Canada. We went through much of the same gag-ordering climate-denying idiocy under Steven Harper (a personal friend of trump by the way). We learned those lessons and wont be deleting anything anytime soon.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by zachrose on Wednesday January 25, 2017 @11:06PM (#13488610)
Seems like this is one of the many things that IPFS would be great for. In practice, I'm curious if the user base is wide enough to be as durable as a real archive. (Although I guess if a few people commit to pinning* the research it might be?)

*Not sure if this is the right term.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by Dylan16807 on Thursday January 26, 2017 @03:28AM (#13489702)
A huge amount of rarely accessed data? IPFS isn't going to guarantee any copies actually exist in that case, and could provide a false sense of security.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by rakoo on Thursday January 26, 2017 @12:37PM (#13493355)
It doesn't make content magically more available, for sure, but it makes backuping a web site trivial, to the point where you can't have content disappear out of sheer will (the only way to have this is to make people uninterested in the content)
Re:Public appeal to backup Climate Monitoring and Diagnostic... by JumpCrisscross on Thursday January 26, 2017 @02:09AM (#13489343)
Wouldn't IPFS just make finding mirrored content easier? It still needs to live somewhere.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by toomanybeersies on Wednesday January 25, 2017 @09:28PM (#13488226)
Bittorrent is a good place to put things, obviously you still need seeds, but it ensures anyone can access it.

Otherwise, I guess AWS Glacier if you want to pay for it.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by astrodust on Wednesday January 25, 2017 @09:47PM (#13488302)
Torrents are a good stepping stone to a more permanent archive but they fundamentally depend on seeders. Likewise Glacier depends on Amazon not being raided by the Feds. I know that sounds totally insane, but we're living in totally insane times.

It needs to be on hardware an organization like the Internet Archive controls or at least ordains. We're talking bunkers in Sweden or Iceland or some country with the political backbone to defend this sort of scientific data.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by steveax on Wednesday January 25, 2017 @09:50PM (#13488311)
usenet.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by philipashlock on Wednesday January 25, 2017 @10:48PM (#13488535)
Data.gov is also coordinating with Archive.org and similar efforts to have a more systematic comprehensive archiving approach. More info should be available soon at http://www.data.gov/developers/harvesting and https://archive.org/details/data_gov
Re:Public appeal to backup Climate Monitoring and Diagnostic... by smkellat on Wednesday January 25, 2017 @10:05PM (#13488374)
Any particular reason we absolutely must DDoS this server at the moment?

From a current session:

ftp> open aftp.cmdl.noaa.gov

Connected to aftp.cmdl.noaa.gov.

421 There are too many connected users, please try later.

ftp>

Re:Public appeal to backup Climate Monitoring and Diagnostic... by bane on Wednesday January 25, 2017 @10:15PM (#13488409)
One of the beautiful things about ftp servers is that you can set a connection limit and they'll just refuse connections above that limit until a slot frees up.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by IgorPartola on Wednesday January 25, 2017 @11:43PM (#13488799)
You can do the same with almost all services I know. Well, at least connection-oriented ones.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by tsomctl on Wednesday January 25, 2017 @08:29PM (#13487970)
Also note https://github.com/climate-mirror/datasets/issues?q=is%3Aiss...

Seems like archive.org should be archiving this stuff, so it's all in one place and future researchers don't have to go hunting all over the internet for it.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by pimlottc on Wednesday January 25, 2017 @08:54PM (#13488085)
Jason Scott (twitter handle @textfiles) is already on it:

https://twitter.com/textfiles/status/824104225688981504

Re:Public appeal to backup Climate Monitoring and Diagnostic... by bane on Wednesday January 25, 2017 @09:36PM (#13488260)
Jason Scott et al may find themselves in line for a medal of freedom at some point in the future.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by niftich on Wednesday January 25, 2017 @09:45PM (#13488294)
Things to keep in mind:

- In the US, the CFAA has language that may or may not be relevant [1]. This isn't intended to be FUD, although I can understand if you think so. As with most legal things, don't believe me -- a random guy -- on the internet; ask a lawyer, or at the very least be aware of your risks.

- Check license and terms, if included. Reddit post claims these [2] are terms, seems okay, but I have not confirmed this for myself.

- FTP is unencrypted and non-tamperproof, so data integrity and data authenticity, and connection privacy isn't guaranteed. And because the original source did not publish checksums over a secure channel, there is no way to know whether these files are the originals. It will be difficult to prove the provenance and accuracy of the data, despite having lots of third-party copies, without the original source having safeguarded the strong cryptographic checksums the entire time, or having published them over a tamperproof channel.

[1] https://www.law.cornell.edu/uscode/text/18/1030 [2] https://www.reddit.com/r/DataHoarder/comments/5q4xxe/erik_fi...

Re:Public appeal to backup Climate Monitoring and Diagnostic... by jMyles on Wednesday January 25, 2017 @10:27PM (#13488463)
> strong cryptographic checksums

They needn't even be strong, right? Even a weak checksum on a data set of this size is basically collision-proof, no?

Re:Public appeal to backup Climate Monitoring and Diagnostic... by niftich on Wednesday January 25, 2017 @10:47PM (#13488531)
It absolutely needs to be a strong cryptographic hash, because you want to protect against both deliberate tampering and accidental modification.

Any other function, whether non-cryptographic, too short, broken or having known weaknesses, while perhaps suitable to detect accidental corruption, will not protect against the former.

For example, SHA-384, or SHA-3 are great choices.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by Dylan16807 on Thursday January 26, 2017 @03:31AM (#13489715)
Though this isn't a case that needs to be append-proof, so SHA-256 is fine.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by jMyles on Wednesday January 25, 2017 @10:57PM (#13488581)
Gotcha. TIL.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by richard_todd on Wednesday January 25, 2017 @10:17PM (#13488417)
Making important data "impossible" to destroy/lose/vandalize is sensible on its own merit. The more signed and decentralized data is, the better. You don't have to be against the new US President to think it's a good idea, but if the political climate brings attention to the matter that would be one silver lining anyway.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by barney54 on Wednesday January 25, 2017 @09:14PM (#13488170)
Why is there any threat of these databases going away?
Re:Public appeal to backup Climate Monitoring and Diagnostic... by astrodust on Wednesday January 25, 2017 @09:17PM (#13488181)
If you're asking that question you haven't been paying attention.

Science is the enemy. It will be destroyed if it continues to get in the way of the administration.

It happened here (Canada) http://www.macleans.ca/news/canada/vanishing-canada-why-were... and it could happen on an even bigger scale in America. The Conservative party went about cutting funding, slashing archives, burning everything to the ground, sometimes literally. They'd do it with little warning, zero fanfare, and an aggressive timeline. One day you had a climate archive, the next the shredding company had taken care of it.

Those servers aren't free. They depend on budgets and grants to stay running. If that money is cut, those files are gone.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by rhizome on Wednesday January 25, 2017 @09:27PM (#13488223)
Because science is a source of truth.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by trhway on Thursday January 26, 2017 @02:52AM (#13489535)
because they most probably contradict alternative facts.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by jakeogh on Wednesday January 25, 2017 @09:53PM (#13488326)
Won't happen.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by bane on Wednesday January 25, 2017 @09:50PM (#13488310)
The reason I posted a reddit thread is that the discussion is full of many other related discussions.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by Skunkleton on Thursday January 26, 2017 @12:47AM (#13489050)
It would be cool to get this all available via bittorrent. The data could be split up into reasonable sized chunks, and everyone could store a little, and everyone would still have immediate access to the whole thing.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by losvedir on Wednesday January 25, 2017 @09:47PM (#13488301)
Should I know who Erik Ficthner is? Is he an NOAA scientist? From what I heard on NPR this morning and a nytimes piece, the Trump administration doesn't seem to be doing anything unusual. I thought it was customary for the incoming administration to make sure the agency is all on the same page. I'm getting hysteria fatigue, I think.
Re:Public appeal to backup Climate Monitoring and Diagnostic... by eropple on Wednesday January 25, 2017 @10:21PM (#13488437)
The anti-science posture of Trump specifically and Republicans generally is alarming by itself. The steady denials of observable fact by this administration and the immediate scrubbing of executive-controlled public relations pages of factual materials about topics that are very inconvenient for them makes worries of Orwell's memory hole very, very reasonable.

This isn't hysteria. This is insurance. Don't be That Guy.

Re:Public appeal to backup Climate Monitoring and Diagnostic... by JanneVee on Thursday January 26, 2017 @02:08AM (#13489338)
I'll leave this here http://www.vanityfair.com/news/2017/01/why-does-peter-thiel-...

It isn't alarmism to have a contingency plan if everything goes to hell. Much like Peter Thiel has one.