If I had a large amount of data I needed to archive long term and cost effectively, I would archive it to 12 different medias with an 4,8 erasure code, such that if any 4 of the 12 media types are readable, then I can recover the data. I'd choose media like a few types of hard disk (different vendors), DVD's, SD cards, USB memory sticks, tapes.

I would then store those bits of media geographically and politically distributed. And I'd store it with paper documents describing the encoding, the file formats, the compression, any encryption, etc. I'd also include a few physical computers (eg. a raspberry pi or laptop) that has all necessary software to read, decode, and display the data. Set it up to be usable by a non-expert - in 1000 years time, there may be nobody who knows how to use a shell or open a file!

And I'd have a 2nd copy of the whole lot on hard drives connected to the internet for day to day serving of the data to people who need to see it. All the stuff above is only needed in case of organisational failure, war, civilisation collapse, etc.

That sounds all nice... but do you actually do that? I'm sure you have some amount of data (maybe not so large) that you want to backup long-term? As most of us do?

If you do that, I would really love to read some more details on how you actually organize that.

My personal data I have no need to keep beyond my own lifespan, and I don't have much of it, so it's easy.

The above is what I have set up for some organisations who want to keep data for thousands of years.

There are other bits to the process, like every 10-30 years, repeat the process with the new data and the old data. This time, the 'old' data will be much smaller compared to the storage mediums, so keep that data uncompressed, preferably unencrypted, and un-erasure coded in every geographic location. That removes many barriers to access the data, and increases the chances someone that finds it in 200 years bothers recovering the data.

Sadly in the future world there is a high chance some of the data is copyright, illegal knowledge or gdpr-impacted and all records need to be erased. There isn't really a good solution to that. It's almost impossible to protect against future humans wanting your data gone.

Maybe Github's "code vault" would be interesting for you: https://github.com/github/archive-program/blob/master/GUIDE.... https://archiveprogram.github.com/

SD cards, usb sticks, and SSDs basically use the same tech where there's a mostly isolated capacitor that holds a charge. Since it's not fully isolated, there's going to be charge leakage when it's powered off. I think most estimates say that you'll start seeing errors and issues after about 3-5 years of being powered off. Not a great option for long term archival storage given the inherent time limits on storage life.

But the leakage rate is very very dependent on temperature. Store it somewhere cool and a 5 year lifetime might be extended to 500 years.

With the right tools you can read back the data and see how much degredation had occurred (even though all the data was readable). Using that info, you can have a good chance at extrapolating the time till the data becomes unreadable.

Yes, but storing many TBs on several low volume disks is a PITA unless you can invest in a robotic library.

I wonder if Sony's ODA format could ever become more popular in the consumer market. I've never heard anybody mention it before.

Alternatively, I wonder if there could even be a "prosumer" robotic library system for common optical disks, something like a desktop archival data jukebox...

There are many examples of cheap self-build robotic systems (basically robotic CD changers). E.g.:

https://hackaday.com/tag/cd-changer/

http://hackalizer.com/jack-the-ripper-is-an-automated-diy-di...

http://hackedgadgets.com/2006/06/07/cd-changing-lego-robot/

Yes, you definitely want sth like that. And further extend it.

Magneto-optical disks using glass media (instead of plastic) have a rated stable media lifetime of at least 50 years and can probably last a century or longer. Glass DVDs are a thing and often can be read in regular DVD drives.

> Magneto-optical disks using glass media

Are there any commercial products that use this technology?

Not magneto, but m-disk makes a blu ray that lasts 1k years supposedly. Some people don't trust it though.

I believe mag-op has fallen out of fashion. I worked with HP-branded mag-op "platters" and drives back in the late 2000's. Plasmon and Sony both had offerings in that space too.

I have anecdotal evidence that CDs can lose 2x2mm chips of the reflective layer after 10-20 years. Also I have seen Blurays written 5 years ago starting to get read very slow because of alot of error correction and read retries going on. You really have to write all Blurays at 1x speed or whatever is lowest and then check using Opti Drive Control by Erik Deppe (good luck finding a compatible drive in 2022!) the quality of your burn. 25/50 GB at a time.

Tape like LTO 6 is quite cheap these days and will last a long time when stored vertically in a dry, cool, dark place. Also at 2.5 TB net capacity per tape with barely used tape prices of 5-10 bucks one can easily write a couple more tapes just in case one is damaged. I think for leader pin damage there is a repair kit and if tape rips in two I remember there was a repair service by IBM which can fix even that and send you a fresh tape with the data or a repaired tape.

I am also interested in some long-term archiving: in particular, .ISOs of various Blu-Ray, DVD, and CD media releases.

Still, aside from it being prohibitively expensive (LTO-8 seems like something of a floor given the size of Blu-Rays), tape backups seem to be a hard area to get into. I did some crappy little DLTs in the 1990s but nothing since, so "what software?" and the like questions are all new to me. And this would be with just a single drive, not even a library.

Iirc the cost per Tb, compared to tape, made discs unviable for most backup/ archival applications.

Depends on who's asking. Amazon Glacier never formally disclosed their storage medium (at least as of a few years ago) and one of the theories on what it might be was actually a robotic optical disc changer library based on BD-XL, and the cost/capacity actually does math out. Yeah, discs might be $15 a pop (for a quad layer/128GB disc) for you as a consumer, but when you're Amazon and you'll be buying the complete output of at least one optical disc factory, the economies of scale kick in. It's just expensive because there's no market for 128GB media for consumers (and honestly these days hardly any market for WORM media at all as a consumer), it's not inherently that expensive to make the discs.

(I believe the final consensus pointed to arrays of HDDs where most of them are powered off, and the number of "live" drives per rack is bounded to allow high density/low cost, hence the need for access time/service level bounds, but the BD-XL idea is still intriguing!)

With the consumer discs, even considering cost per GB, the amount of effort required to handle a large library of low-capacity discs is just too great even if the cost is a little bit better. 128GB discs would have been very usable 5 years ago but again, those discs were never affordable to consumers, and the 25GB was still some effort at that time. Today even 128GB is not all that much, as data has grown. As far as I know there is nothing realistic on the horizon to replace blu-ray with higher capacity either, if movie content started being released in 8K it probably would be something like BD-XL with AV1 encoding (or maybe H265 again), not a fundamentally new iteration like DVD->BD.

The future for consumer storage seems to be SSDs and hard drives for fast and slow/bulk storage, and cloud for nearline storage. Tape is still relevant for enterprises though especially in automatic libraries.

Interesting, i didn't know about glacier.

The theory of hard drives being shut off/ powered on dynamically in a rack sounds intriguing. Sounds simple and yet difficult because of the rare usecase, i.e. no commodity hardware available. Maybe something to test out for colo backups to keep power usage down and prolong disk health.

I believe that commodity hardware is sort of available. Look at MAID hardware. It's aimed at enterprise customers so it's pretty expensive if you buy new but there's probably older stuff on ebay or a reseller site.

Capacity per disc too. Blu-Ray discs tops at 125GB and there’s no cheap and easy way to automate disc handling to work around that.