Re:Google releases snappy, the compression library used in B...
by ot on Tuesday March 22, 2011 @06:37PM
(#2356865)
I wonder if they had evaluated LZO (http://www.oberhumer.com/opensource/lzo/) before writing this. It is quite well-tested (a variant on it runs in the Mars Rovers) and very very fast: the author reports 16MB/sec on a Pentium 133, on modern architectures it should easily get to the 500MB/sec claimed by snappy.
[ Parent
| on HN ]
Re:Google releases snappy, the compression library used in B...
by lemming on Tuesday March 22, 2011 @07:12PM
(#2357008)
"The LZO algorithms and implementations are copyrighted OpenSource distributed under the GNU General Public License."
Does this even make sense? Can you apply the GPL to an algorithm? As I understand it, if there's no patent I should be able to implement it with no problems.
Re:Google releases snappy, the compression library used in B...
by wmf on Tuesday March 22, 2011 @07:22PM
(#2357052)
People who want to upsell you a commercial version tend to say such things.
Re:Google releases snappy, the compression library used in B...
by bhickey on Tuesday March 22, 2011 @07:41PM
(#2357119)
Be warned: the main source code in the 'src' directory
is a real pain to understand as I've experimented with
hundreds of slightly different versions. It contains
many #if and some gotos, and is *completely optimized
for speed* and not for readability. Code sharing of the
different algorithms is implemented by stressing the
preprocessor - this can be really confusing. Lots of
marcos and assertions don't make things better.
Given the author's statements, I don't know that I'd feel comfortable using LZO in a production environment.
Re:Google releases snappy, the compression library used in B...
by malkia on Tuesday March 22, 2011 @11:56PM
(#2357924)
We used it in our games since Ultimate Spiderman. I've got assembly version working on the PS2 IOP chip (33Mhz, the chip used to run PS1 games, otherwise I/O and sound for PS2).
I've got 5mb/s decompressed data speed, so that was speeding our disk access. I had to tweak just a little bit the source code and make sure unaligned writes (4 bytes) were used and that made it x2 or x3 faster.
Re:Google releases snappy, the compression library used in B...
by drv on Tuesday March 22, 2011 @07:25PM
(#2357064)
The README says:
"In our tests, Snappy usually is faster than algorithms in the same class (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression ratios."
Re:Google releases snappy, the compression library used in B...
by ot on Tuesday March 22, 2011 @07:26PM
(#2357074)
Interesting, I had missed that. It also says
> Finally, snappy can benchmark Snappy against a few other compression libraries (zlib, LZO, LZF, FastLZ and QuickLZ), if they were detected at configure time.
I'll do some tests, I'm curious about the results
Re:Google releases snappy, the compression library used in B...
by tonfa on Tuesday March 22, 2011 @07:41PM
(#2357118)
Please post the results :)
Re:Google releases snappy, the compression library used in B...
by wolf550e on Wednesday March 23, 2011 @11:08PM
(#2362651)
~/snappy-read-only $ ./snappy_unittest -norun_microbenchmarks -lzo testdata/*
testdata/alice29.txt :
LZO: [b 1M] bytes 152089 -> 82691 54.4% comp 64.5 MB/s uncomp 206.4 MB/s
SNAPPY: [b 4M] bytes 152089 -> 90965 59.8% comp 171.5 MB/s uncomp 375.8 MB/s
testdata/asyoulik.txt :
LZO: [b 1M] bytes 125179 -> 73217 58.5% comp 52.5 MB/s uncomp 173.2 MB/s
SNAPPY: [b 4M] bytes 125179 -> 80207 64.1% comp 137.8 MB/s uncomp 301.7 MB/s
testdata/baddata1.snappy :
LZO: [b 1M] bytes 27512 -> 26487 96.3% comp 24.4 MB/s uncomp 491.6 MB/s
SNAPPY: [b 4M] bytes 27512 -> 26675 97.0% comp 305.2 MB/s uncomp 1465.7 MB/s
testdata/baddata2.snappy :
LZO: [b 1M] bytes 27483 -> 26528 96.5% comp 24.3 MB/s uncomp 499.1 MB/s
SNAPPY: [b 4M] bytes 27483 -> 26724 97.2% comp 331.8 MB/s uncomp 1660.4 MB/s
testdata/baddata3.snappy :
LZO: [b 1M] bytes 28384 -> 27380 96.5% comp 24.1 MB/s uncomp 488.2 MB/s
SNAPPY: [b 4M] bytes 28384 -> 27476 96.8% comp 275.7 MB/s uncomp 1346.5 MB/s
testdata/cp.html :
LZO: [b 1M] bytes 24603 -> 11621 47.2% comp 57.2 MB/s uncomp 258.5 MB/s
SNAPPY: [b 4M] bytes 24603 -> 11838 48.1% comp 190.0 MB/s uncomp 443.5 MB/s
testdata/fields.c :
LZO: [b 1M] bytes 11150 -> 4663 41.8% comp 73.8 MB/s uncomp 259.8 MB/s
SNAPPY: [b 4M] bytes 11150 -> 4728 42.4% comp 207.3 MB/s uncomp 431.7 MB/s
testdata/geo.protodata :
LZO: [b 1M] bytes 100000 -> 20423 20.4% comp 115.8 MB/s uncomp 429.9 MB/s
SNAPPY: [b 4M] bytes 100000 -> 23488 23.5% comp 359.0 MB/s uncomp 625.8 MB/s
testdata/grammar.lsp :
LZO: [b 1M] bytes 3721 -> 1781 47.9% comp 66.8 MB/s uncomp 311.4 MB/s
SNAPPY: [b 4M] bytes 3721 -> 1800 48.4% comp 214.9 MB/s uncomp 461.2 MB/s
testdata/house.jpg :
LZO: [b 1M] bytes 126958 -> 127173 100.2% comp 20.2 MB/s uncomp 1420.8 MB/s
SNAPPY: [b 4M] bytes 126958 -> 126803 99.9% comp 2037.6 MB/s uncomp 7578.7 MB/s
testdata/html :
LZO: [b 1M] bytes 102400 -> 21027 20.5% comp 115.1 MB/s uncomp 423.8 MB/s
SNAPPY: [b 4M] bytes 102400 -> 24140 23.6% comp 362.4 MB/s uncomp 703.5 MB/s
testdata/html_x_4 :
LZO: [b 1M] bytes 409600 -> 82980 20.3% comp 120.9 MB/s uncomp 416.4 MB/s
SNAPPY: [b 4M] bytes 409600 -> 96472 23.6% comp 359.0 MB/s uncomp 698.0 MB/s
testdata/kennedy.xls :
LZO: [b 1M] bytes 1029744 -> 357315 34.7% comp 133.5 MB/s uncomp 533.6 MB/s
SNAPPY: [b 4M] bytes 1029744 -> 425735 41.3% comp 294.1 MB/s uncomp 432.6 MB/s
testdata/kppkn.gtb :
LZO: [b 1M] bytes 184320 -> 71671 38.9% comp 84.3 MB/s uncomp 235.9 MB/s
SNAPPY: [b 4M] bytes 184320 -> 70535 38.3% comp 231.7 MB/s uncomp 373.8 MB/s
testdata/lcet10.txt :
LZO: [b 1M] bytes 426754 -> 221290 51.9% comp 57.6 MB/s uncomp 181.9 MB/s
SNAPPY: [b 4M] bytes 426754 -> 243710 57.1% comp 153.7 MB/s uncomp 341.4 MB/s
testdata/mapreduce-osdi-1.pdf :
LZO: [b 1M] bytes 94330 -> 76999 81.6% comp 24.9 MB/s uncomp 810.9 MB/s
SNAPPY: [b 4M] bytes 94330 -> 77477 82.1% comp 709.0 MB/s uncomp 1669.5 MB/s
testdata/plrabn12.txt :
LZO: [b 1M] bytes 481861 -> 294610 61.1% comp 51.4 MB/s uncomp 164.8 MB/s
SNAPPY: [b 4M] bytes 481861 -> 329339 68.3% comp 129.8 MB/s uncomp 281.9 MB/s
testdata/ptt5 :
LZO: [b 1M] bytes 513216 -> 86232 16.8% comp 119.0 MB/s uncomp 506.1 MB/s
SNAPPY: [b 4M] bytes 513216 -> 93455 18.2% comp 479.8 MB/s uncomp 670.8 MB/s
testdata/sum :
LZO: [b 1M] bytes 38240 -> 17686 46.2% comp 57.8 MB/s uncomp 267.0 MB/s
SNAPPY: [b 4M] bytes 38240 -> 19837 51.9% comp 194.5 MB/s uncomp 401.0 MB/s
testdata/urls.10K :
LZO: [b 1M] bytes 702087 -> 309320 44.1% comp 55.7 MB/s uncomp 265.1 MB/s
SNAPPY: [b 4M] bytes 702087 -> 357267 50.9% comp 216.9 MB/s uncomp 499.3 MB/s
testdata/xargs.1 :
LZO: [b 1M] bytes 4227 -> 2450 58.0% comp 56.3 MB/s uncomp 286.2 MB/s
SNAPPY: [b 4M] bytes 4227 -> 2509 59.4% comp 173.8 MB/s uncomp 397.3 MB/s
This is on a Core2, 64bit, GCC 4.5.2. Please someone port it to C and linux-kernel code so it could be used with zram instead of LZO.