News for hackers. Stuff that matters.
Re:Pack: A New Container Format for Compressed Files by OttoCoddo on Saturday March 23, 2024 @12:23PM (#39801017)
It was hard to believe for me, too. And I didn't stumble upon it; I looked for it closely, and that was a point in the note. People did not look properly for nearly three decades. Many things have changed, but we computer people are still using the same tools. I am not saying old is not good; the current solutions are great, but what are we, if we don't look for the better?

Yes it is that much faster, and a good part of it is because of the multi-thread design, but as a reminder, WinRAR or 7-Zip are too multi-thread, and you can see the difference. To satisfy your doubt, I suggest running Pack for yourself. I am looking for more data on its behaviour on different machines and data.

Can I ask why do you need a version without ZSTD? If you are thinking that compression slows it down, I should say no. Pack is the first of its kinds that "Store" is slowing it down. Because its compression is smart, it will skip any non-compressible content.

On the same machine and the same Linux source code test:

Pack: 194 MB, 1.3 s

Pack (With no Press): 1.25 GB, 1.8 s

Re:Pack: A New Container Format for Compressed Files by out_of_protocol on Saturday March 23, 2024 @04:35PM (#39802929)
Pure zstd (or .tar.zstd) vs pack vs patched 7z+zstd would be more interesting, how much overhead introduced by pack format itself - in size and speed
Re:Pack: A New Container Format for Compressed Files by OttoCoddo on Saturday March 23, 2024 @04:54PM (#39803040)
I answered this question here: https://news.ycombinator.com/item?id=39801083 If that is not enough, let me know.
Re:Pack: A New Container Format for Compressed Files by out_of_protocol on Saturday March 23, 2024 @06:16PM (#39803436)
tar.zst vs pack is looking great, thanks! Also there is https://github.com/mcmilk/7-Zip-zstd

.pack vs zst-7z with the same compression settings would b interesting. That will be pure container overhead

Re:Pack: A New Container Format for Compressed Files by xcdzvyn on Sunday March 24, 2024 @08:45AM (#39806874)
My concern with Pack obliging me to compress is that compression becomes less pluggable; I'd much rather my archive format be agnostic of compression, as with tar, so that I can trivially move to a better compression format when one inevitably comes to be.
Re:Pack: A New Container Format for Compressed Files by OttoCoddo on Sunday March 24, 2024 @09:41AM (#39807170)
You got a point. Although with that that option comes a great cost: We will lose portability, speed and even reliability.

Portability: Receiver (or future you) needs to know what you used, and what version even.

Speed: If you want to do the archive part first (tar) and then compress (gz), you will get much lower speed (as shown in the note).

Reliability: Most people use tar with gz anyway, but if you use it with not so popular algorithm and tools, you will risk having a file that may or may not work into the future.

Pack plan is to use the best of time (Zstandard) and if an update is needed in years to come, it will add support for the new algorithm updates. All Pack clients must only write the latest version (and read all previous versions) and that makes sure almost all use the best of their time.