Re:Hop: Faster than unzip and tar at reading individual files
by iainmerrick on Wednesday November 10, 2021 @06:41PM
(#29182495)
Have you looked at using LZ4 compression? Supposedly it has a very fast decompressor, so it could be better than uncompressed files for your purposes.
[ Parent
| on HN ]
Re:Hop: Faster than unzip and tar at reading individual files
by Jarred on Wednesday November 10, 2021 @07:15PM
(#29182774)
I don't think builtin compression is necessary, but it would be better if the CLI supported it similarly to tar.
Compressing a 57.5 MB .hop archive with `gzip -c` returns an 11.7 MB file. This is a similar compression ratio as tar -> tar.gz
Compression is great for sending files over the network, but for fast reads, it might be undesirable
Re:Hop: Faster than unzip and tar at reading individual files
by iainmerrick on Wednesday November 10, 2021 @07:37PM
(#29182984)
It’s worth benchmarking -- LZ4 can decompress faster than SSD read speeds, so it could well be a net time saving.
Re:Hop: Faster than unzip and tar at reading individual files
by eby on Wednesday November 10, 2021 @08:10PM
(#29183245)
How about compressing the files while building the archive instead of compressing the entire archive itself? That ought to preserve reasonably fast reads. It won't give optimal compression but in practice should do alright (and of course will be smaller than no compression).