Re:Ask HN: How would you store 10PB of data for your startup...
by simcop2387 on Sunday April 25, 2021 @12:05AM
(#26930282)
Best I know of for that is something like lrzip still, but even then it's probably not state of the art. https://github.com/ckolivas/lrzip
It'll also take a hell of a long time to do the compression and decompression. It'd probably be better to do some kind of chunking and deduplication instead of compression itself simply because I don't think you're ever going to have enough ram to store any kind of dictionary that would effectively handle so much data. You'd also not want to have to re-read and reconstruct that dictionary to get at some random image too.
Re:Ask HN: How would you store 10PB of data for your startup...
by water8 on Saturday April 24, 2021 @09:10PM
(#26929534)
A movie is a series of similar images and while it does allow temporal compression in a 3rd axis to the 2d raster, H265 is about as good as it gets at the moment but its also lossy which might not be tolerable.
I'm not sure what the state of lossless algorithms might be for that though.