Re:Apt 2.0
by julian-klode on Sunday March 08, 2020 @07:49AM
(#22517572)
Like, zstd -21 --ultra requires a whopping 365MB of RAM to compress a 16 MiB file, zstd -19 requires 12 MB, and xz requires 10 MB.
Decompressing -21 requires 20MB vs 10MB for -19 and xz.
zstd is _way_ faster at decompressing though, even from --ultra: some 13x faster in the case of Arch Linux.
Re:Apt 2.0
by julian-klode on Tuesday March 10, 2020 @04:47PM
(#22539523)
I know that, but compression time is not as relevant as you think it is. If you look back at the measurements we did 2 years ago:
https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/04...
You can see that the performance gain from zstd under realistic scenarios is about 10%.
you can see that switching from xz to zstd only improved firefox install time from 37s to 33s; running in eatmydata to avoid fsync from dpkg improved it to 12.5s (8.5s with zstd).
I don't know what Arch linux does, but they likely do not correctly fsync() data after its been written to a temporary file, and then fsync the directory after files have been renamed to their final names, as is required to achieve consistent results in a crash.
But then with deltas, mirror sizes grows even more, which is a hard sell.
Arch Linux gets away with using zstd -21 but that's not practical for general purpose Linux distributions, as it has significant memory requirements compared to xz.
Also the speedup in practice is negligent - while zstd is much faster than xz, most of the time installing packages is actually spent in fsync(). So you only see huge speedups when run in eatmydata.