News for hackers. Stuff that matters.
Re:Tape still beats SSDs and hard drives when it comes to pr... by jarfil on Tuesday September 24, 2019 @10:08PM (#21066877)
Nothing. They estimate an average compression based on an average dataset with uncompressed text data. That wouldn't work for things like already compressed video, audio, encrypted data, and so on.
Re:Tape still beats SSDs and hard drives when it comes to pr... by userbinator on Tuesday September 24, 2019 @11:38PM (#21067374)
I really wish they wouldn't add stuff like compression and so forth, because I feel like it's an "abstraction layer violation" and it's misleading in a way that the majority of other storage media isn't; for example, a 1TB hard drive is going to store (at least) 10^12 bytes regardless of what those bytes contain. A 60 minute audio tape will always record 60 minutes of pure silence, white noise, or anything in between. Yet somehow data tape is marketed with a theoretical compressed capacity.
Re:Tape still beats SSDs and hard drives when it comes to pr... by nightfly on Wednesday September 25, 2019 @02:24AM (#21068222)
It's super annoying, and it's annoying to (attempt) to disable the hardware compression on the tape drives as well. We've got a tape library with 2 two LTO7 tape drives, and even though we're accessing it through the linux device file that should be accessing the tapes in an un-compressed mode diagnostic tools indicate that compression is still be used when writing to tape. Since we're pre-compressing our data with gzip having the tape drive try and apply compression is a wasted effort and I wonder if we're losing write speed to it.
Re:Tape still beats SSDs and hard drives when it comes to pr... by cl0ckt0wer on Tuesday September 24, 2019 @10:24PM (#21066975)
works great on giant uncompressed database backups though
Re:Tape still beats SSDs and hard drives when it comes to pr... by sliken on Tuesday September 24, 2019 @10:27PM (#21066991)
Sure, or just datadump | gzip > /dev/tape or similar. Tape loves to brag about bandwidth/capacity and then hide the assumed 2.5x compression in a footnote.
Re:Tape still beats SSDs and hard drives when it comes to pr... by namibj on Tuesday September 24, 2019 @10:40PM (#21067071)
Please consider zstd instead. It is superior in every way except memory usage and compatibility with old systems.
Re:Tape still beats SSDs and hard drives when it comes to pr... by erik_seaberg on Wednesday September 25, 2019 @01:38AM (#21067948)
zstd looks good but it's only three years old, so use cases need to fit in the lifetime of today's platform. gzip is about to turn 27 and everyone has heard of it, which makes me pretty confident in finding decoders after another decade or two.

(Actually this problem is why I haven't owned a tape drive for a while.)

Re:Tape still beats SSDs and hard drives when it comes to pr... by sliken on Tuesday September 24, 2019 @10:47PM (#21067101)
Sure. Anything, not like text compression is hard.
Re:Tape still beats SSDs and hard drives when it comes to pr... by nightfly on Wednesday September 25, 2019 @02:25AM (#21068225)
pigz!
Re:Tape still beats SSDs and hard drives when it comes to pr... by terrelln on Wednesday September 25, 2019 @06:41PM (#21076385)
Zstd also has a multithreaded mode. On the CLI `zstd -T0` tells zstd to use all available cores.
Re:Tape still beats SSDs and hard drives when it comes to pr... by namibj on Thursday September 26, 2019 @07:36AM (#21079958)
Please be aware that this will significantly impact compression ratio. So much actually, that I don't dare use it by default. I'd often use the non-multithreaded mode as for some data (perf.data files with large stack captures, most recently) and reasonable compression speeds of >=50MB/s input multi-threaded modes are not pareto-optimal.

I'll get a small benchmark script for that done over lunch. Come back here later.

Re:Tape still beats SSDs and hard drives when it comes to pr... by Cyan4973 on Thursday September 26, 2019 @06:11PM (#21086132)
> Please be aware that this will significantly impact compression ratio.

Is this a continuation from previous message ? Because multithreading mode of zstd doesn't impact its compression ratio. It's actually an interesting property of zstd : whatever the nb of threads used, the compression ratio remains the same (aka. reproducible).

Re:Tape still beats SSDs and hard drives when it comes to pr... by jolmg on Tuesday September 24, 2019 @10:48PM (#21067109)
Before reading up on LTO, I thought maybe it was some kind of physical compression where it would write the data closer together at the cost of read/write speed or something (doesn't really make sense in retrospect). I'm kind of disappointed that it was just regular data compression with an assumed ratio. :/
Re:Tape still beats SSDs and hard drives when it comes to pr... by perl4ever on Wednesday September 25, 2019 @05:54PM (#21075987)
I spend most of my time these days with a database that seems to compress to 4% of its size...