News for hackers. Stuff that matters.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by CodesInChaos on Thursday May 25, 2023 @07:37AM (#36069436)
For me it's crazy how tape vendors always advertise "compressed capacity" which is 2.5 times the actual capacity. Imagine a laptop containing a 2TB SSD getting advertised as containing a "5TB SSD"...
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by mschuster91 on Thursday May 25, 2023 @07:48AM (#36069520)
That's because of different expectations. For a tape drive, you don't care about latency, power efficiency, random access or write amplification, all that matters is getting as much data as possible on a tape for the lowest amount of investment into tapes. So the tapes are advertised with the amount of net data that you can fit onto it, assuming a standard distribution of compressible vs non-compressible data.

For a laptop SSD, all of these matter. You can't do much compression because compression consumes a lot of power and latency upon access, some compression schemes make random access harder (i.e. compression schemes without an index where you have to scan through intermediate checkpoints or, in the worst case, sequentially through the entire media), and it may lead to write amplification as well if you need to add a piece of data in the middle of a file (basically the issue with shingle HDDs). As a result, no compression possible, and so SSDs are advertised with the raw capacity (or, in fact, they are underadvertised because SSDs need spare block capacity to account for wear).

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by f1refly on Thursday May 25, 2023 @07:50AM (#36069542)
But it completely disregards that most files you'd want to backup are already compressed! Even if I where to turn on aggressive compression for my ssd despite all the inconveniences, it wouldn't change much. I'll have to agree with the op, storage capacity marketing for tape is weird if not outright a scam.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by jl6 on Thursday May 25, 2023 @08:31AM (#36069934)
A lot of tape usage is in mainframe and traditional big iron systems where data is bulk unloaded from databases in a fairly raw format, and it’s convenient to let the tape infrastructure handle both compression and encryption.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by HPsquared on Thursday May 25, 2023 @07:49AM (#36069530)
Compressibility depends on the data though, one person might be storing log files while another stores compressed video.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by dabiged on Thursday May 25, 2023 @08:44AM (#36070043)
I had a lto4 (max capacity 800gb) last year that took over 5tb of data. The data was ASCII log files from a server that had been running for years. Lots of repetition (time stamps, everything ok messages, you have mail messages) and the drive compression ate it for breakfast.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by loeg on Thursday May 25, 2023 @10:10AM (#36070948)
> assuming a standard distribution of compressible vs non-compressible data.

There's no such thing as a standard distribution of compressible data. Customer data varies wildly.

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by londons_explore on Thursday May 25, 2023 @07:54AM (#36069571)
A lot of business data will be encrypted by policy before it gets anywhere near the (perhaps offsite) backup system. That means any compression the tape drive offers will be useless.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by chongli on Thursday May 25, 2023 @09:19AM (#36070366)
Why not compress the data before encrypting it? Seems like a really easy to implement solution that would speed up backup and recovery and cost almost nothing to implement.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by teraflop on Thursday May 25, 2023 @09:28AM (#36070447)
Yes of course, but that still means the compression offered by the tape drive is ineffective, even though that compression is assumed as part of the advertised tape capacity.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by Karunamon on Thursday May 25, 2023 @09:53AM (#36070728)
Even last generation tape drives have hardware encryption built in. And it doesn't make sense to me that they would encrypt, then compress. That would completely invalidate the compression for absolutely no benefit.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by count on Thursday May 25, 2023 @10:55AM (#36071504)
The benefit being they can sell more tapes...
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by londons_explore on Thursday May 25, 2023 @11:43AM (#36072203)
Nobody trusts encryption built into tape drives for the same reason nobody trusts encryption built into hard drives.

Even things like Windows Bitlocker and LUKS/dmcrypt on Linux totally ignore the drives ability to do any encryption, and do all encryption using the CPU before the drive sees the data.

Bitlocker/LUKS could easily just calculate the encryption keys and send them to the drive and trust it to encrypt/decrypt data for them... but they don't.

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by MichaelZuo on Thursday May 25, 2023 @12:39PM (#36072924)
> Nobody trusts encryption built into tape drives for the same reason nobody trusts encryption built into hard drives.

Source? Last I heard many companies rely on the encryption supplied of tape drive systems.

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by adrian_b on Thursday May 25, 2023 @01:51PM (#36073828)
It would be more accurately to say that all those who understand how encryption works (which excludes most people in managerial positions, even if they have the power to decide what kind of encryption is used in their companies) cannot trust any kind of encryption embedded in a hardware device whenever it has to be used for protecting really important data.

The reason is that all encryption is based on the separation of place between the encrypted data and the encryption key. Whoever can access the encrypted data must not have any way to access the encryption key.

Whenever you give your encryption key to a hardware device, or worse, when the hardware device also generates itself the encryption key, it becomes impossible to ensure that the attacker will not be able to access the encryption key.

It is impossible to know how the encryption keys are stored inside a hardware device and how and when they are erased and how easy or how difficult it will be in the future for an attacker to retrieve them.

It is impossible to believe any marketing claim of the vendor of a hardware encryption device about how tamper-resistant the device is, because such claims have very frequently been proven to be lies (even when the claims come from the largest companies, e.g. Microsoft and many others like it) and it is too difficult to distinguish truth from lies in such cases.

The only reliable means of encryption are in software, under complete end user control (or equivalently, in a custom FPGA).

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by MichaelZuo on Thursday May 25, 2023 @02:16PM (#36074099)
So can you link to a source?
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by throwanma on Thursday May 25, 2023 @06:08PM (#36076943)
At some point, knowledge comes from experience, ^ he is my source.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by MichaelZuo on Thursday May 25, 2023 @06:34PM (#36077249)
It appears adrian_b is a pseudonym, not even linked to a real identity, how can that be a source?
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by wizofaus on Thursday May 25, 2023 @04:25PM (#36075724)
Surely if the device automatically decrypts data as you read from it without having to supply a key or passcode of some sort then you're not protecting it against anything! It may automatically manage/store the actual keys used to encrypt the data, but there must be some mechanism to ensure decryption is not permitted until you supply some external form of credential (which ideally is cryptographically required to access the keys - if not then obviously there's a risk the device's controller code could be reverse engineered to bypass the credential check).
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by Karunamon on Thursday May 25, 2023 @01:15PM (#36073387)
Considering that industry standard software like backup exec has built-in support for hardware encryption (which is actually part of the LTO standard), I would need to see a source for "nobody trusts encryption built into tape drives"
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by jandrese on Thursday May 25, 2023 @12:42PM (#36072954)
Using the encryption provided by the drive seems like a terrible idea. What happens in 10 years when you want to retrieve the data and that company has gone out of business?
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by throw0101c on Thursday May 25, 2023 @01:00PM (#36073184)
As I understand it (AIUI) the encryption is in the standard:

* https://www.lto.org/encryption/

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by Karunamon on Thursday May 25, 2023 @01:19PM (#36073445)
It's functionality exposed by the drive as part of the tape standard, not something unique to vendor of the drive. Any LTO-aware software given the same key would be able to decrypt the data using any drive that can read that generation of tape.

That's probably one of the items on the list of reasons there are so few vendors in the space, LTO drives are essentially fungible and don't do anything interesting across vendors.

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by rbanffy on Thursday May 25, 2023 @04:10PM (#36075511)
The worst part of compressing at the drive is that you’ll need to transfer 2.5x-ish more data than you’d need if you compressed prior to transferring.

In the end, it’s encrypted, compressed, decrypted, and decompressed multiple times on the way from the platter (or flash) to the tape recording head. It feels kind of dumb, really, but I’ll agree that if the computer doesn’t have the private key of the drive, stealing the data on tape will be a lot harder.

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by cabirum on Thursday May 25, 2023 @09:22AM (#36070393)
Data can be compressed, then encrypted, then shipped offsite?
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by Accujack on Thursday May 25, 2023 @03:22PM (#36074915)
At present it's usually sent to the tape drive which does hardware data compression, hardware encryption, and then writes to the tape so it can be sent off site.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by wang_li on Thursday May 25, 2023 @02:45PM (#36074450)
Typical enterprises will utilize the tape drive to do the encryption. The objective is that your tapes are encrypted when they go off site.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by Accujack on Thursday May 25, 2023 @03:21PM (#36074908)
No, it won't. Just because it's encrypted doesn't mean it's totally random.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by etskinner on Thursday May 25, 2023 @03:52PM (#36075300)
It does, actually. Most encryption schemes have the property of ciphertext indistinguishability: https://en.wikipedia.org/wiki/Ciphertext_indistinguishabilit...

If they didn't, they'd be vulnerable: https://en.wikipedia.org/wiki/Distinguishing_attack

However, something close to your point is true. If they compress the data and then encrypt, the ciphertext will indeed be smaller than the original data.

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by Accujack on Thursday May 25, 2023 @10:46PM (#36079145)
The encryption used by e.g. IBM tape drives is two layered - strong/slow and weak/fast. The Asymmetric key encryption used is the "strong" encryption, but it's far too slow to be used to encrypt the whole tape. So IBM encrypts the tape data with the weak/fast symmetric algorithm and then encrypts the key for that with the strong/slow asymmetric one.

If you compress the symmetrically encrypted data, you will indeed gain some capacity. Not as much as you would compressing the raw data, but a visible amount, because the symmetric algorithm doesn't have the property of indistinguishability.

That property is not needed for tape storage because anyone who has the tape can safely assume it contains a ciphertext. The tape has header information saying whether it's encrypted or not.

If you read the links you posted, you will find out that it's not "most" cryptosystems, it's some. Some applications just don't need indistinguishability and LTO Tapes are one of them.

All that said, IBM does compress first then encrypt to make the most of the data compression, but the resulting ciphertext isn't completely random.

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by ajb on Thursday May 25, 2023 @03:53PM (#36075304)
If your encryption is any good, it means that it's indistinguishable from random if you don't know the key. Which the tape drive won't.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by rbanffy on Thursday May 25, 2023 @04:04PM (#36075444)
While you are technically correct - the best type of correct - the encrypted data should be indistinguishable from random data. If it’s not, you need better cryptography, as pointed out by the sibling comments.
Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by Accujack on Thursday May 25, 2023 @10:47PM (#36079163)
Actually, it's not that simple. Read my response above.

You can't really say indistinguishability makes one cryptosystem better than another because it prevents distinguishing attacks because for some applications it's both not needed and computationally expensive.

Re:Tape Storage Trundles On, Increases Yearly Volume to 128 ... by Accujack on Thursday May 25, 2023 @03:20PM (#36074902)
Tape drives like LTOs use some fairly good hardware compression... believe it or not, it's significantly better than the average zip program. Depending on the data sent to the tape, it can achieve 50-60% reduction in size.