News for hackers. Stuff that matters.

Show HN: Indexed_bzip2 2.0 – I used AI to make bzip2 decompression 4x faster

https://github.com/mxmlnkn/librapidarchive/releases/tag/v2.0.0

Hi HN,

I started indexed_bzip2 years ago as a seekable, parallelized bzip2 backend for ratarmount. It has been dormant for a while and seemed almost finished. Out of interest, I tried looking for bugs with DeepSeek V4.1 Flash, which found dozens of them, most with a minimal reproducing test. After reviewing and merging those and making a bugfix release, I prompted it with: "Speed up decompression by 2x". Surprisingly, it worked! You can try it out yourself with (timings on Ryzen 3900X):

    pip install indexed_bzip2==2.0  # Depending on your default: --uploaded-prior-to P0D, or uvx, pipx
    
    # single-threaded gzip decompression
    time igzip  -d -c      silesia-20x.tar.gz  | cksum  # 7.5 s
    time  gzip  -d -c      silesia-20x.tar.gz  | cksum  # 22.4 s
    
    # single-core bzip2
    time  bzip2 -d -c      silesia-20x.tar.bz2 | cksum  # 105 s
    time lbzip2 -d -c -n 1 silesia-20x.tar.bz2 | cksum  # 77.6 s
    time ibzip2 -d -c -P 1 silesia-20x.tar.bz2 | cksum  # 23.7 s
    
    # parallelized
    time lbzip2 -d -c      silesia-20x.tar.bz2 | cksum  # 6.7 s
    time ibzip2 -d -c      silesia-20x.tar.bz2 | cksum  # 3.8 s
It not only added lots of dynamically dispatched SIMD code, ABI3 wheels, Trusted Publishing, all things I had long dreaded setting up, but also found libsais for an early inverse-BWT speedup. A contributor then used Claude Fable to replace libsais with an interleaved lane-walking inverse-BWT and SIMD symbol kernels, which hides the random pointer-chase latency and gave another 1.7x.

Code: https://github.com/mxmlnkn/librapidarchive/

ReadMe: https://github.com/mxmlnkn/librapidarchive/blob/HEAD/python/...

Benchmarks: https://github.com/mxmlnkn/librapidarchive/blob/HEAD/results...

( no comments | on HN )

There are no comments yet.