powturbo

TurboBench: The Ultimate Data Compression Benchmark 100+ Codecs. The Powerhouse in a Single Executable — The Compression Lie Detector

Tired of misleading benchmarks with I/O overhead, cache effects, and CPU throttling? TurboBench delivers high-precision, zero-overhead results you can actually trust.

Why TurboBench Stands Out The only benchmark that includes 100+ popular, latest, and fastest compressors (LZ77, Rolz, BWT, Context Mixing, Entropy Coders, and more). 100% in-memory testing — no disk I/O to skew your results. Supports multiple files, recursive directories, and smart concatenation of small files into multiblock tests. Built to avoid common pitfalls like cache scenarios and throttling found in other tools. Unlock peak performance — TurboBench reduces throttling by intelligently controlling the CPU frequency Automatic sorting, updating, and merging of result files. A nightly github release is created automatically whenever submodules change. Creates a separate result file ([email protected]) for each compressor set TurboBench rank aggregation crowns the top compression libraries with — see who's king!

Try TurboRC using 16-bits format: https://github.com/powturbo/Turbo-Range-Coder

Thank for your analyse. It seems it's a corner case, benchmarking a base64 encoded random file is not a typical case. igzip has no more advantage above libdeflate. It has only fast compression at the levels 0,1,2 but with mediocre compression ratio.

EDIT: I've seen the file you're referencing is 4Gi-Base64. This file is not very compressible (75% with gzip). It's possible that igzip is simply storing the file or some parts of it without compression. This explain why it can be faster that libdeflate, because in this case igzip is using memcpy at decompression.

Sorry, a digit was missign, but now https://github.com/powturbo/TurboBench/issues/43

Well, a correct benchmarking is not done with special data, but with datasets that represent a large set of distributions. Such datasets are for ex. einwik8/9 for text, silesia for a mixed dataset. If you provide a link for a dataset 10-100MB, I can verify your claims, because I'm not aware of a dataset where igzip is 2 times faster than libdeflate. In TurboBench there is not I/O involved, additionally it's single threaded.

libdeflate compress better and has faster decompression than igzip.

See the silesia single core in-memory benchmark here [1] comparing zlib,libdeflate,igzip,...

https://github.com/powturbo/TurboBench/issues/4

file html8 : 100MB random html pages from a 1m Alexa Top sites corpus.

Number of pages = 1178

Average length = 84886 bytes

CPU: AMD 7840HS 3.8-5.1GHZ

You can use TurboBench [1] to benchmark libdeflate against zlib, igzip, zlib-ng and others.

Download TurboBench from Releases [2]

Here Some Benchmarks:

- https://github.com/zlib-ng/zlib-ng/issues/1486

- https://github.com/powturbo/TurboBench/issues/43

[1] https://github.com/powturbo/TurboBench

[2] https://github.com/powturbo/TurboBench/releases

** Cython bindings for Turbo Base64 [1] **

- 20-30x faster than the standard library

- Benchmarks faster than any other C base64 library

- Fastest implementation of AVX, AVX2, and AVX512 base64 encoding

- No other dependencies

[1] - https://github.com/powturbo/Turbo-Base64

You can use TurboBench [1] to benchmark igzip, zlib-ng and others.

Download TurbBench from Releases [2]

Here Some Benchmarks:

- https://github.com/zlib-ng/zlib-ng/issues/1486

- https://github.com/powturbo/TurboBench/issues/43

[1] https://github.com/powturbo/TurboBench

[2] https://github.com/powturbo/TurboBench/releases

Yes, especially SIMD Neon where gcc producing horrible Neon code for all versions < gcc-12 even by using simd intrinsics. From version 12 gcc is at same level as clang.

You can try it : https://github.com/powturbo/Turbo-Base64

> Obviously it's not impossible, since it's happening on my laptop. Well, zstd cli is more optimized for multithreading and asynchrounous i/o. zstd is overlapping cpu decompression with reading/writing. Additionally zstd compressed data is more dense. It's possible that on your notebook you have lz4 decompression including i/o slower that zstd. This is not the way lz are compared in general. Otherwise nobody will use lz4 anymore.

> I haven't tried TurboBench, but this is what I get when running the built-in lz4 and zstd benchmarks You can see, even on your machine lz4 is faster than zstd. Now it's obvious that asynchrounous i/o + multithreading is making the difference in the cli case.

> However, I don't understand exactly what this metric means. Does it mean that lz4 can ingest the input at ~4 GB/s or that it produces the output at ~4 GB/s? In data compression the metrics are always relative to the original uncompressed size.

> But in any case, what matters to me as a user is how quickly I can compress and decompress a file. Well, this depends of the use case. I'm using 7zip (GUI) and it's working fine for me.

It's very hard to make benchmarks when multithreading or i/o is involved. You'll get often different results on different hardware configurations and the number of threads.

"In memory" means without i/o (disk,ssd,network), only in RAM. Streaming means compressing/decompressing in chunks and can also be "IN memory".

No, No and No. It's algorithmically impossible that zstd is decompressing faster than lz4 by using the same environment.

You are comparing the zstd with asynchrounous I/O + multithreading against a single thread simple lz4 cli.

Additionally you're using some exotic and not representative hardware.

In theory, the only single thread decompression case where zstd can beats lz4 is when you have slow (reading) i/o.

lz4 & zstd are mostly used as libraries and real benchmarks are not considering I/O or multithreading.

This is what I'm getting with TurboBench & enwik9. lz4 decompressing 2,8x faster than zstd.

      C Size  ratio%     C MB/s     D MB/s   Name            
   356828015    35.7     400.71    1557.59   zstd 1 
   416874817    41.7     312.31    1967.66   lzav         
   509199084    50.9     630.78    4395.08   lz4 1           

Download TurboBench [1] from [2] and make the tests on your machine.

[1] - https://github.com/powturbo/TurboBench

[2] - https://github.com/powturbo/TurboBench/releases

lz4 is ALWAYS decompressing faster (> 2x) thand zstd

LZAV now integrated into TurboBench [1]

Download TurboBench for windows from releases [2]

Lenovo IdeaPad 5 Pro - Ryzen 6600HS - DDR5 6400 - gcc-13.1

File: silesia.tar 220MB (Mixed text/binary)

    C Size  ratio%     C MB/s     D MB/s        Name     
  77885207    36.7      43.20    4479.24        lz4 9          
  81343184    38.4     123.17    4299.68        lz4 3 
  85664894    40.4     324.11    5171.12        lzturbo 11 
  88672448    41.8     435.96    2769.77        lzav v1.3
  99553305    47.0     914.25    5425.54        lzturbo 10
 100883728    47.6     813.76    4667.85        lz4 1
 211948544   100.0   16032.92   16023.24        memcpy

lzturbo (closed source) included as reference

[1] https://github.com/powturbo/TurboBench

[2] https://github.com/powturbo/TurboBench/releases

see also Discussion with a brotli developer [2] and "Chrome: Implement zstd content-encoding" [3] :

[1] - Benchmark App: https://github.com/powturbo/TurboBench

[2] - https://twitter.com/powturbo/status/1678333893626195969

[3] - https://bugs.chromium.org/p/chromium/issues/detail?id=124697...

Thanks for the confirmaation. If your query engine can't exploit fine grained access, then general purpose compressors like lz4 or zstd with large compressed blocks are the better choice for numerical data, especially in combination with transpose (+delta).

Most of time series databases (ex. DuckDB) are storing floating point data as 64 bits.

They are reporting some extraordinary compression ratio by using a gorilla/chimp like algorithm.

However as shown in this benchmark, lot of time series data (ex. temparature, climate data, stocks,...) don't have more than 1 or 2 fixed decimal digits and can be stored losslessly in 16/32 bits integers.

Integer compression algorithms can then be used, which results in significant compression ratio and several times faster than the gorilla like algorithms.

TurboGorilla, the fastest Gorilla (or chimp) based algo in c, cannot exceed 1GB/s in decompression, wherea TurboPFor is in the order of 10Gb/s, TurboBitByte is >100Gb/s.

There is nothing to earn from non GPL license, maybe some small reputation and more stars on github, but that's all. Contribution is minimal if any, and when then you have more work. Dual license is better so others can't complain, they can't use your library because of GPL.

- Turbo-Base64 is used in clikhouse

- Turbo-Base64 is available in archlinux:https://aur.archlinux.org/packages/turbo-base64

- 100% C (C++ headers), as simple as memcpy

- No other base64 library encode or decode faster

- Scalar can be faster than other SSE or ARM Neon based base64 libraries

- SSE faster than other SSE/AVX/AVX2! base64 library

- Fastest AVX2 implementation TurboBase64 AVX2 decoding up to ~2x faster than other AVX2 libs.

- TurboBase64 is 3-4 times faster than other libs for short strings

- Fastest ARM Neon base64

- (2023.04) avx512 - 2x faster than avx2, faster than any other implementation

- Dynamic CPU detection and JIT scalar/sse/avx/avx2/avx512 switching

- Base64 robust error checking, optimized for long+short strings

- Portable library, 32/64 bits, SSE/AVX/AVX2/AVX512, ARM Neon, Power9 Altivec

- OS:Linux amd64, arm64, Power9, MacOs+Apple M1, s390x. Windows: Mingw, visual c++

- Big endian + Little endian

- Ready and simple to use library, no armada of files, no hassles dependencies