News for hackers. Stuff that matters.
Re:Ask HN: How would you store 10PB of data for your startup... by d110af5ccf on Saturday April 24, 2021 @01:24AM (#26922480)
On the contrary, batches of images with a high degree of similarity compress _very_ well. You have to use an algorithm specifically designed for that task though. Video codecs are a real world example of such - consider that H. 265 is really compressing a stream of (potentially) completely independent frames under the hood.

I'm not sure what the state of lossless algorithms might be for that though.

Re:Ask HN: How would you store 10PB of data for your startup... by simcop2387 on Sunday April 25, 2021 @12:05AM (#26930282)
Best I know of for that is something like lrzip still, but even then it's probably not state of the art. https://github.com/ckolivas/lrzip

It'll also take a hell of a long time to do the compression and decompression. It'd probably be better to do some kind of chunking and deduplication instead of compression itself simply because I don't think you're ever going to have enough ram to store any kind of dictionary that would effectively handle so much data. You'd also not want to have to re-read and reconstruct that dictionary to get at some random image too.

Re:Ask HN: How would you store 10PB of data for your startup... by water8 on Saturday April 24, 2021 @09:10PM (#26929534)
A movie is a series of similar images and while it does allow temporal compression in a 3rd axis to the 2d raster, H265 is about as good as it gets at the moment but its also lossy which might not be tolerable.
Re:Ask HN: How would you store 10PB of data for your startup... by papageek on Monday April 26, 2021 @09:15AM (#26942178)
H266 VVC looks impressive. Waiting to get my hands on fpga codec for testing.
Re:Ask HN: How would you store 10PB of data for your startup... by cerved on Wednesday April 28, 2021 @04:29AM (#26967066)
right but we're not talking about compressing a video stream but compressing individually compressed pictures, big difference