News for hackers. Stuff that matters.
Re:Apt 2.0 by julian-klode on Sunday March 08, 2020 @07:49AM (#22517572)
Like, zstd -21 --ultra requires a whopping 365MB of RAM to compress a 16 MiB file, zstd -19 requires 12 MB, and xz requires 10 MB.

Decompressing -21 requires 20MB vs 10MB for -19 and xz.

Re:Apt 2.0 by m0zg on Sunday March 08, 2020 @08:32PM (#22521812)
zstd is _way_ faster at decompressing though, even from --ultra: some 13x faster in the case of Arch Linux.

https://en.wikipedia.org/wiki/Zstandard

Re:Apt 2.0 by julian-klode on Tuesday March 10, 2020 @04:47PM (#22539523)
I know that, but compression time is not as relevant as you think it is. If you look back at the measurements we did 2 years ago:

https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/04...

You can see that the performance gain from zstd under realistic scenarios is about 10%.

you can see that switching from xz to zstd only improved firefox install time from 37s to 33s; running in eatmydata to avoid fsync from dpkg improved it to 12.5s (8.5s with zstd).

I don't know what Arch linux does, but they likely do not correctly fsync() data after its been written to a temporary file, and then fsync the directory after files have been renamed to their final names, as is required to achieve consistent results in a crash.