News for hackers. Stuff that matters.
Re:LTO Tape data storage for Linux nerds by londons_explore on Thursday January 27, 2022 @09:14AM (#30100788)
If I had a large amount of data I needed to archive long term and cost effectively, I would archive it to 12 different medias with an 4,8 erasure code, such that if any 4 of the 12 media types are readable, then I can recover the data. I'd choose media like a few types of hard disk (different vendors), DVD's, SD cards, USB memory sticks, tapes.

I would then store those bits of media geographically and politically distributed. And I'd store it with paper documents describing the encoding, the file formats, the compression, any encryption, etc. I'd also include a few physical computers (eg. a raspberry pi or laptop) that has all necessary software to read, decode, and display the data. Set it up to be usable by a non-expert - in 1000 years time, there may be nobody who knows how to use a shell or open a file!

And I'd have a 2nd copy of the whole lot on hard drives connected to the internet for day to day serving of the data to people who need to see it. All the stuff above is only needed in case of organisational failure, war, civilisation collapse, etc.

Re:LTO Tape data storage for Linux nerds by albertzeyer on Thursday January 27, 2022 @09:52AM (#30101255)
That sounds all nice... but do you actually do that? I'm sure you have some amount of data (maybe not so large) that you want to backup long-term? As most of us do?

If you do that, I would really love to read some more details on how you actually organize that.

Re:LTO Tape data storage for Linux nerds by londons_explore on Thursday January 27, 2022 @10:10AM (#30101453)
My personal data I have no need to keep beyond my own lifespan, and I don't have much of it, so it's easy.

The above is what I have set up for some organisations who want to keep data for thousands of years.

There are other bits to the process, like every 10-30 years, repeat the process with the new data and the old data. This time, the 'old' data will be much smaller compared to the storage mediums, so keep that data uncompressed, preferably unencrypted, and un-erasure coded in every geographic location. That removes many barriers to access the data, and increases the chances someone that finds it in 200 years bothers recovering the data.

Sadly in the future world there is a high chance some of the data is copyright, illegal knowledge or gdpr-impacted and all records need to be erased. There isn't really a good solution to that. It's almost impossible to protect against future humans wanting your data gone.

Re:LTO Tape data storage for Linux nerds by raron on Thursday January 27, 2022 @12:30PM (#30103383)
Re:LTO Tape data storage for Linux nerds by scheme271 on Friday January 28, 2022 @12:50AM (#30111081)
SD cards, usb sticks, and SSDs basically use the same tech where there's a mostly isolated capacitor that holds a charge. Since it's not fully isolated, there's going to be charge leakage when it's powered off. I think most estimates say that you'll start seeing errors and issues after about 3-5 years of being powered off. Not a great option for long term archival storage given the inherent time limits on storage life.
Re:LTO Tape data storage for Linux nerds by londons_explore on Friday January 28, 2022 @01:54AM (#30111466)
But the leakage rate is very very dependent on temperature. Store it somewhere cool and a 5 year lifetime might be extended to 500 years.

With the right tools you can read back the data and see how much degredation had occurred (even though all the data was readable). Using that info, you can have a good chance at extrapolating the time till the data becomes unreadable.