Show HN: Indexed_bzip2 2.0 – I used AI to make bzip2 decompression 4x faster
https://github.com/mxmlnkn/librapidarchive/releases/tag/v2.0.0
Hi HN,I started indexed_bzip2 years ago as a seekable, parallelized bzip2 backend for ratarmount. It has been dormant for a while and seemed almost finished. Out of interest, I tried looking for bugs with DeepSeek V4.1 Flash, which found dozens of them, most with a minimal reproducing test. After reviewing and merging those and making a bugfix release, I prompted it with: "Speed up decompression by 2x". Surprisingly, it worked! You can try it out yourself with (timings on Ryzen 3900X):
pip install indexed_bzip2==2.0 # Depending on your default: --uploaded-prior-to P0D, or uvx, pipx
# single-threaded gzip decompression
time igzip -d -c silesia-20x.tar.gz | cksum # 7.5 s
time gzip -d -c silesia-20x.tar.gz | cksum # 22.4 s
# single-core bzip2
time bzip2 -d -c silesia-20x.tar.bz2 | cksum # 105 s
time lbzip2 -d -c -n 1 silesia-20x.tar.bz2 | cksum # 77.6 s
time ibzip2 -d -c -P 1 silesia-20x.tar.bz2 | cksum # 23.7 s
# parallelized
time lbzip2 -d -c silesia-20x.tar.bz2 | cksum # 6.7 s
time ibzip2 -d -c silesia-20x.tar.bz2 | cksum # 3.8 s
It not only added lots of dynamically dispatched SIMD code, ABI3 wheels, Trusted Publishing, all things I had long dreaded setting up, but also found libsais for an early inverse-BWT speedup. A contributor then used Claude Fable to replace libsais with an interleaved lane-walking inverse-BWT and SIMD symbol kernels, which hides the random pointer-chase latency and gave another 1.7x.Code: https://github.com/mxmlnkn/librapidarchive/
ReadMe: https://github.com/mxmlnkn/librapidarchive/blob/HEAD/python/...
Benchmarks: https://github.com/mxmlnkn/librapidarchive/blob/HEAD/results...
(
no comments
| on HN
)
There are no comments yet.