They say that they have a 50 year lifespan, but everyone in the industry knows that's a blatant lie these days. The tape customers transfer everything over to the new generation every 2 to 3 years, so it's never tested and none of the customers really care.
You also only get a couple hundred writes to a tape in it's lifetime. That includes bulk appends as the stress from seeking is a major factor of what kills them. They really need to be treated as WORO media, bulk, the whole 12TB or what have you at once. Not a great fit for home use.
These days the main benefit to tape is the physical safety of having the media be inside a cartridge, and the streaming read speeds are stupid fast so you can reconstruct a backup faster.
Home use, even with a data hoarder sized NAS, is better served with a bluray burner with multiple copies.
These tape-drives have 300MB/s+ of read/write speed. Sequential yes, but almost all backup tasks are sequential.
Optical has 10MB/s or so. You get more out of a 100Mbps connection to (insert cloud storage provider here), let alone Gbps.
At a minimum, a modern, reasonable mechanism for backups needs to be faster than cloud (100Mbps or Gbps), otherwise its basically worthless to the consumer. Hard drives and Tape get there, but there was no real way to improve Optical's read/write performance (outside of overengineered "jukebox" robots available to Facebook and a few other select groups), so it went fully obsolete.
I just gave a look at the first Blu Ray media on Newegg.com. Here's what is search #1: https://www.newegg.com/verbatim-6x-25gb-bd-r/p/N82E168171301...
That's 6x speed, or roughly 25MB/s. You can store 25GBs per disk and it is going to take 1000% longer than a $150 5TB Hard Drive AND you're going to have to sit there and remove / add a new disk every few minutes.
And for what? The hard drive is cheaper after all of that.
And yeah, you'd be buying 16x 128GB media in this use case. Not the bottom of the barrel stuff for giving home movies to grandma.
So push the "ZFS Scrub" button every 6 months.
Don't store hard drives. Store a NAS. The entire freaking computer is stored as a unit. Every few months, turn it on, push "ZFS Scrub" to double-check the data, and you're set.
BluRays also degenerate over time unless stored in proper UV-sealed conditions / temperature controlled conditions. Everything requires a degree of maintenance and checking. The question is how to best automate that checking process.
Hard drives are read/written to at 200+ MB/s, making these maintenance checks much faster. They're also bigger, which means no need to manually insert / remove disks from drives. This entire process could be automated from a "Wake on LAN" packet and a few clicks from any terminal (phone, computer, whatever).
-------
I dunno. I'm looking up these BD-R XL drives you're talking about: its like $30 per 100GB disk or something. That's an obscene price for so little storage. I guess if all your data fits in one of those disks its fine, but... I got some archived movies and stuff on my NAS (50GB per Blu Ray). Needed to store some of my video editing files (so I need the original data that matches with my video editing files).
The real data of importance of mine probably fits in a BluRay. But then I won't have all the other stuff I've saved up "because I can" on my 2x 5TB NAS (Mirrored, so only 5TB of storage).
3 copies, 2 different media, 1 offsite.
I've seen large orgs do what you're saying and lose nearly everything because one bug took them out completely.
Ehh? Not really. All of these things we're talking about are components of our backup strategy.
If I wanted an offsite backup, its just rsync to some cloud provider (like rsync.net) or something. I don't do that because I don't think my data is worth the recurring cost of that, but its an option.
-------
My point is that when my solution is 200MB/s and on the order of $500 to $1000 for the component ($500 if you build your own NAS, $1000 if you buy premade parts. Assuming 2x 5TB hard drive is just $300 for 5TBs of _mirrored_ storage)...
While your proposed component costs $30 per 0.1TB and read/writes at a lousy 70MB/s... to get the equivalent mirrored setup you need to buy 100 Blu-Rays or roughly $3000 in Blu Ray XLs alone (2 copies of your data across 2-different Blu Rays, for the same redundancy factor as the 2x Hard Drive solution).
So I have to raise my eyebrows a little bit. How are you checking that the data doesn't degrade? Are you manually checking all of those BluRays you've created for reliability? That's a lot of sitting around and inserting / removing drives.
Its literally cheaper to build a 2nd NAS and stick more hard drives into it, and keep that 2nd NAS offsite somewhere.
-------
If you're gonna hold my feet to the fire over this "2-media" thing, then my 2nd media of choice would be Flash storage before Optical. Because SPEED is king. Speed means you can checksum your data and ensure that your backups are still good. I think Flash is a bit expensive compared to HDD, but based off of these BL-XL disks, I'm thinking that Flash is actually cheaper than Blu Ray and something like 10,000% faster. (Tapes would be a more ideal 2nd media... but I'm not "big enough" to make tapes cost-effective).
Yes, checksums and scrubbing. If you want to protect against bitrot (in ANY medium), you MUST double-check your backups over time.
TEST your backups. Any backup strategy that doesn't have a regular testing schedule is null-and-void in my opinion.
I seem to think that HDD, then Flash, and MAYBE Tape (if you're going really, really big) are the mediums of choice of the modern computer user. I'm not really seeing where Optical fits in today's world. Maybe a future Disk format (with maybe TB-level disks) can make Optical a thing again... but 100GB Blu Rays aren't really in the works for the modern user.
70MB/s is slow. And 100GB is too small per storage unit.
Again, tapes are for archival storage. Not nearline, not live, archival. As in, you write the data and likely don't come back to it for months to years.
Some of the issues you need to address:
- Trust. How can someone trust you with potentially proprietary / copyrighted data?
- Speed. Is it faster to copy the data to a tape locally, then to transfer to you?
- Reliability. How can the client be sure that you have made a successful backup?
- Advertising. Did you reach your target base to make them aware of your product and address all their concern to make them consider it?
Trust: yep, hard to say for this one. At least I can provide actual guarantees (for instance, none of our storage is out of the country).
Speed: it's possible to send us tapes directly. However the main selling point of our solution is to allow people to upload relatively small volumes of data continuously, then archive it all to tape in large batches when you have enough to fill an LTO.
Reliability: the interface allows the user to see where their files are (on which tape), to run checksum controls on tapes, and restore from tape. Every tape is sold with 3 free operations a year (archive, checksum verification, restoration).
Advertising: that's clearly not our strong point :)
If you don't do that, eventually you'll get to a point where it can't repair anything, and then you gain nothing over using a filesystem that doesn't do this, is the point.
Objections on the failure rate of HDDs are absolutely valid, but then one should also need to consider the bigger picture (e.g. storage loss), in which case, having a remote copy is also important.
You have an interesting argument - by the same reasoning, when one has eventually filled up a 12 TB HDD, it would no longer be hoarding by tomorrow's standard. In other words, at this point, one should be able to get the next generation of HDDs for cheap, and it's this fact that makes all tape drives unnecessary.
Now I wonder whether it was a mistake for me to buy a spare 14 TB HDD to 1:1 mirror my new file server for cold backup. Perhaps a smaller one should be good for 5 years anyway...
You're basing your argument on extreme and unrepresentative assumptions.
First of all, you do not need to fill a storage device to the brim to justify it's use. You can use the exact same rationale to claim no one needs 500GB HDs even though they are pretty much standard these days.
Additionally, you falsely assume that your data storage needs start the very moment someone buys a drive, and up till then they have no data lying around. That's not the case at all. People buy high-density storage devices because they already have the data lying around, and they don't want to lose it. You're ignoring that people already have piles of CDRs/DVDs/Blu-ray disks lying around.
Additionally, you are somehow assuming that people would buy storage devices for their density, ignoring their primary use case: long term data preservation.
I cheerfully quantify just about everything in my life, yet somehow I missed that one. I am a little bit of a data hoarder, or maybe just a little paranoid. Seeing those numbers is actually quite helpful. Thank you.
PS I get downvoted a lot for comments like this, so in case it sounds sarcastic or facetious, it is not. I mean it sincerely.
Indeed. The home directory on my PC at home has 80GB of files stored. Most of it is just there out of laziness. About the only thing I try to keep backups of are my tax returns.
What do people have on a home machine that needs enterprise-grade backups?
You might say photos and videos. I guess that's a personal thing; I realized a long time ago that I never spent any time actually looking at any of the pictures I took, so I stopped taking them.
That is a completely logical take. However, I sort of think of it as a posterity move. Although my kids might not particularly want to see pictures of themselves, their great grand children—or beyond—might love it. I would pay a pretty penny to see home movies of my grandparents, whom I never knew.
I realized in the last year or so that the only digital media I have which I would be genuinely sad to lose were wedding photos - so I saved those to 3 different cloud providers, made a Blu-ray copy, and a copy on the NAS. If I lose all of that in some tragedy, chances are I've got bigger things to worry about.
It adds up quite fast. And being able to have put everything on a tape every year with a label on it would give me some peace of mind, even more if I place 2 of them on different location.
I think the market for tape will always be the same few giant companies for whom it will be priced just a bit less than the next cheapest option.
The problem was, nobody actually backs up so not enough people bought tapes/drives so they quit making them.
During the 80s there was no digital consumer camera market. Nowadays every person can easily generate hundreds of megabytes of photos and videos per day. Each hour of a 4K video can be close to 7GB, and we're already seeing cellphones which are able to record 4K video at 60fps and 1080p videos at 240fps.
On the Mac side, in 1989 you could apparently buy a Mac IIci for $6200 with 1MB or 4MB, "expandable to 128MB".
I know your comment was a throwaway idea but I got a little bit too involved looking up information around this answer - so I've committed further and looked a little deeper to avoid looking too stupid: Currently with BaFe (Barium Ferrite) as their magnetic particle, LTO-7+ tapes have particle sizes less than 100nm in size (https://indico.cern.ch/event/637013/contributions/2669089/ - check out slides 14 and 15 in the powerpoint.).
VHS on the other hand is a little more tricky to find information on I suspect due to the age of the technologies involved it's going to be significantly bigger - but the closest I've found to a technical document is a student paper at NYU which refers to this IEEE Paper (https://ieeexplore.ieee.org/document/50474) saying that the average particle size for VHS is ~300nm and for S-VHS at ~150nm.
But that first presentation also mentions smoothness increases over time with LTO, which suggests improvements in the coating process.
I suspect improvements in the coating, combined with both the size of magnetic particle shrinking, enabled an increase in density of particles on the tape. This in conjunction with improved thinness of the tapes enabled large increases in tape length for the same cartridge size (nearly double the length from LTO-1 -> LTO-8), has in turn led to the enormous jumps in capacity we've seen in LTO.
Meaning unfortunately I don't think it'll be possible to do super interesting things with VHS.
OK, but OP said:
>It's really a pain to keep transferring your old backups every 2-5 years...
If the OP really needs dozens of TBs of capacity every few year, they definitely don't fit in the home user market they are talking about.
With a new HD you can just copy it across in one step.
rsync --archive --acls --xattrs --checksum --vvv /src /destYou're going to use the drive for 50 years? 50 years? Half a century? And you think it'll be working and supporting a format with a useful capacity in half a century?
If anything, ZFS makes that whole system you just described easier and more automatic. Its not really hard for me to push the "scrub" button on my Nas4Free box every few months.
Money - The cost factor of buying the new HDD. And to be safe, you need to have more than one medium. That's again more cost.
Time - Ensuring that your existing machine is compatible with both your old and new disks (IDE vs SATA or working DVD drives for example).
I don't recommend backing up to HDD's. They are prone to early failure from portable use because of vibration and drops.
What we need are more affordable sata SSD's. Currently nvme SSD's are very close to the same price.
If you could sell a 1TB portable usb 3/usb c backup device that comes with good software the vast majority of people would be set.
Your backup device should stay at home, on the desk or in the closet where you hide your networking equipment. There's very little benefit to trying to carry around your backup device with your laptop on a regular basis, and if you do, you still need another backup device that isn't going to get stolen at the same time as your laptop.
This. Even 100MB/s SSD will do. We need a type of NAND and SSD that offers High Capacity and low speed and low cost. The Current 2TB is ~$170, compared to a 2TB Portable HDD at ~$60.
And I am wondering, if Silent File Corruption, bitrot etc are things of the past on SSD? Does BTRFS / ZFS even make sense on SSD?
[1] https://www.amazon.com/Green-1TB-Internal-SSD-WDS100T2G0A/dp...
What I really want is an affordable Tape storage for us common folks / home users to archive data long term. It's really a pain to keep transferring your old backups every 2-5 years to a new CD / DVD / portable HDD. There is a market for home users that really don't want to put their data on the "cloud". And this generation is really creating so much data, some of which, I am sure they would like to store long-term.