News for hackers. Stuff that matters.
Re:Zlib-ng: a performance-oriented fork of zlib by gjm11 on Friday June 05, 2015 @07:38AM (#9664900)
It seems like this would be more compelling if accompanied by some benchmarks showing how much better the performance is than the original zlib's.
Re:Zlib-ng: a performance-oriented fork of zlib by moreati on Friday June 05, 2015 @08:18AM (#9665021)
A very rough and ready, impromptu benchmark.

zlib latest took 26 seconds to compress 648 MB of text, zlib-ng latest took 16 seconds.

    alex@martha:~/src/zlib$ export TESTFILE=../enwiki-20141106-pages-articles-multistream-index.txt
    alex@martha:~/src/zlib$ du -h $TESTFILE
    648M	/home/alex/src/enwiki-20141106-pages-articles-multistream-index.txt
    alex@martha:~/src/zlib$ LD_LIBRARY_PATH=./ time ./minigzip <$TESTFILE >/dev/null
    23.75user 0.17system 0:26.05elapsed 91%CPU (0avgtext+0avgdata 1864maxresident)k
    56472inputs+0outputs (0major+144minor)pagefaults 0swaps
    alex@martha:~/src/zlib$ LD_LIBRARY_PATH=./ time ./minigzip <$TESTFILE >/dev/null
    24.29user 0.17system 0:26.80elapsed 91%CPU (0avgtext+0avgdata 1772maxresident)k
    0inputs+0outputs (0major+141minor)pagefaults 0swaps
    alex@martha:~/src/zlib$ cd ../zlib-ng
    alex@martha:~/src/zlib-ng$ LD_LIBRARY_PATH=./ time ./minigzip <$TESTFILE >/dev/null
    14.66user 0.11system 0:16.16elapsed 91%CPU (0avgtext+0avgdata 1604maxresident)k
    256inputs+0outputs (0major+140minor)pagefaults 0swaps
    alex@martha:~/src/zlib-ng$ LD_LIBRARY_PATH=./ time ./minigzip <$TESTFILE >/dev/null
    14.64user 0.12system 0:16.09elapsed 91%CPU (0avgtext+0avgdata 1840maxresident)k
    0inputs+0outputs (0major+144minor)pagefaults 0swaps
The machine is a MBP 11,2 with an i7-4750HQ CPU @ 2.00GHz, running Ubuntu 15.04 x64.
Re:Zlib-ng: a performance-oriented fork of zlib by nly on Friday June 05, 2015 @09:12AM (#9665251)
What about decompression? Are we already I/O bound? Compression is irrelevant in many cases, where you can precompress static content.
Re:Zlib-ng: a performance-oriented fork of zlib by moreati on Friday June 05, 2015 @10:03AM (#9665553)
zlib latest takes 3 seconds, zlib-ng takes 13 seconds

    alex@martha:~/src/zlib$ du -sh $TESTFILE
    197M	/home/alex/src/enwiki-20141106-pages-articles-multistream-index.txt.gz
    alex@martha:~/src/zlib$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null
    3.36user 0.03system 0:03.87elapsed 87%CPU (0avgtext+0avgdata 1632maxresident)k
    8inputs+0outputs (0major+90minor)pagefaults 0swaps
    alex@martha:~/src/zlib$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null
    3.34user 0.04system 0:03.76elapsed 90%CPU (0avgtext+0avgdata 1704maxresident)k
    0inputs+0outputs (0major+91minor)pagefaults 0swaps
    alex@martha:~/src/zlib$ cd ../zlib-ng/
    alex@martha:~/src/zlib-ng$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null
    12.20user 0.04system 0:13.63elapsed 89%CPU (0avgtext+0avgdata 1576maxresident)k
    264inputs+0outputs (1major+86minor)pagefaults 0swaps
    alex@martha:~/src/zlib-ng$ LD_LIBRARY_PATH=./ time ./minigzip -d <$TESTFILE >/dev/null
    12.15user 0.08system 0:13.58elapsed 90%CPU (0avgtext+0avgdata 1428maxresident)k
    0inputs+0outputs (0major+84minor)pagefaults 0swaps
Re:Zlib-ng: a performance-oriented fork of zlib by gjm11 on Friday June 05, 2015 @10:33AM (#9665734)
So the optimizations in zlib-ng enable it to decompress ... more than 4x slower than vanilla zlib? That seems worthy of some further investigation.
Re:Zlib-ng: a performance-oriented fork of zlib by pimlottc on Friday June 05, 2015 @01:12PM (#9666636)
I'm curious, was your decompression test file compressed with zlib? zlib-ng? Does it make a difference?
Re:Zlib-ng: a performance-oriented fork of zlib by Dead2 on Friday June 05, 2015 @01:24PM (#9666707)
Unfortunately the minigzip implementation used is using a very minimal codebase to implement the functionality, among these is a problem where it will read and decompress 1 byte per call to inflate(). A patch and pull request for this has been available since yesterday at least, and I merged this today.

In stock zlib this codepath is only used if you defined Z_SOLO, but in zlib-ng it is the default unless you run configure with --zlib-compat (or otherwise set the correct defines).

Please retest with the updated commit, or even better with --zlib-compat if you want to get close to an apples-to-apples comparison.

EDIT: PS: This goes to show that minigzip can not really be used as a reliable benchmark, since actual applications linking to the library could be using it differently (for better or worse).

Re:Zlib-ng: a performance-oriented fork of zlib by jsnell on Friday June 05, 2015 @12:09PM (#9666268)
Edit:

Results with zlib-ng and a newer version of the Cloudflare changes:

https://www.snellman.net/blog/archive/2015-06-05-updated-zli...

Original:

Here are some benchmarks of the Intel and Cloudflare forks that this project is based on:

https://www.snellman.net/blog/archive/2014-08-04-comparison-...

The speedups are not insignificant. (I'll see about updating the post to include results for zlib-ng).

Re:Zlib-ng: a performance-oriented fork of zlib by Dead2 on Friday June 05, 2015 @01:27PM (#9666732)
As I mentioned in another comment, the minigzip used is very suboptimal due to reading and decompressing only 1 byte per inflate() call. Please retest with --zlib-compat or the fix that was merged today.
Re:Zlib-ng: a performance-oriented fork of zlib by jsnell on Friday June 05, 2015 @02:00PM (#9666931)
Sure thing, updated.

I'm not in the habit of including compression level 6 in the results, since the application I care about defaults to 5 :) And I wanted a good spread of different compression levels in the results.

Re:Zlib-ng: a performance-oriented fork of zlib by Dead2 on Friday June 05, 2015 @01:36PM (#9666782)
Btw, I always wondered.. Why didn't you test compression level 6, it is the default level, and used by for example stock apache/nginx settings.

Also #zlib-ng on freenode if you or anyone else want to discuss.