News for hackers. Stuff that matters.
Re:Google releases snappy, the compression library used in B... by ot on Tuesday March 22, 2011 @06:37PM (#2356865)
I wonder if they had evaluated LZO (http://www.oberhumer.com/opensource/lzo/) before writing this. It is quite well-tested (a variant on it runs in the Mars Rovers) and very very fast: the author reports 16MB/sec on a Pentium 133, on modern architectures it should easily get to the 500MB/sec claimed by snappy.
Re:Google releases snappy, the compression library used in B... by lemming on Tuesday March 22, 2011 @07:12PM (#2357008)
"The LZO algorithms and implementations are copyrighted OpenSource distributed under the GNU General Public License."

Does this even make sense? Can you apply the GPL to an algorithm? As I understand it, if there's no patent I should be able to implement it with no problems.

Re:Google releases snappy, the compression library used in B... by wmf on Tuesday March 22, 2011 @07:22PM (#2357052)
People who want to upsell you a commercial version tend to say such things.
Re:Google releases snappy, the compression library used in B... by bhickey on Tuesday March 22, 2011 @07:41PM (#2357119)

    Be warned: the main source code in the 'src' directory 
    is a real pain to understand as I've experimented with 
    hundreds of slightly different versions. It contains 
    many #if and some gotos, and is *completely optimized  
    for speed* and not for readability. Code sharing of the 
    different algorithms is implemented by stressing the 
    preprocessor - this can be really confusing. Lots of 
    marcos and assertions don't make things better.
Given the author's statements, I don't know that I'd feel comfortable using LZO in a production environment.
Re:Google releases snappy, the compression library used in B... by malkia on Tuesday March 22, 2011 @11:56PM (#2357924)
We used it in our games since Ultimate Spiderman. I've got assembly version working on the PS2 IOP chip (33Mhz, the chip used to run PS1 games, otherwise I/O and sound for PS2).

I've got 5mb/s decompressed data speed, so that was speeding our disk access. I had to tweak just a little bit the source code and make sure unaligned writes (4 bytes) were used and that made it x2 or x3 faster.

Re:Google releases snappy, the compression library used in B... by drv on Tuesday March 22, 2011 @07:25PM (#2357064)
The README says:

  "In our tests, Snappy usually is faster than algorithms in the same class (e.g. LZO, LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression ratios."
Re:Google releases snappy, the compression library used in B... by ot on Tuesday March 22, 2011 @07:26PM (#2357074)
Interesting, I had missed that. It also says

> Finally, snappy can benchmark Snappy against a few other compression libraries (zlib, LZO, LZF, FastLZ and QuickLZ), if they were detected at configure time.

I'll do some tests, I'm curious about the results

Re:Google releases snappy, the compression library used in B... by tonfa on Tuesday March 22, 2011 @07:41PM (#2357118)
Please post the results :)
Re:Google releases snappy, the compression library used in B... by wolf550e on Wednesday March 23, 2011 @11:08PM (#2362651)

  ~/snappy-read-only $ ./snappy_unittest -norun_microbenchmarks -lzo testdata/*
  testdata/alice29.txt                     :
  LZO:    [b 1M] bytes 152089 ->  82691 54.4%  comp  64.5 MB/s  uncomp 206.4 MB/s
  SNAPPY: [b 4M] bytes 152089 ->  90965 59.8%  comp 171.5 MB/s  uncomp 375.8 MB/s
  testdata/asyoulik.txt                    :
  LZO:    [b 1M] bytes 125179 ->  73217 58.5%  comp  52.5 MB/s  uncomp 173.2 MB/s
  SNAPPY: [b 4M] bytes 125179 ->  80207 64.1%  comp 137.8 MB/s  uncomp 301.7 MB/s
  testdata/baddata1.snappy                 :
  LZO:    [b 1M] bytes  27512 ->  26487 96.3%  comp  24.4 MB/s  uncomp 491.6 MB/s
  SNAPPY: [b 4M] bytes  27512 ->  26675 97.0%  comp 305.2 MB/s  uncomp 1465.7 MB/s
  testdata/baddata2.snappy                 :
  LZO:    [b 1M] bytes  27483 ->  26528 96.5%  comp  24.3 MB/s  uncomp 499.1 MB/s
  SNAPPY: [b 4M] bytes  27483 ->  26724 97.2%  comp 331.8 MB/s  uncomp 1660.4 MB/s
  testdata/baddata3.snappy                 :
  LZO:    [b 1M] bytes  28384 ->  27380 96.5%  comp  24.1 MB/s  uncomp 488.2 MB/s
  SNAPPY: [b 4M] bytes  28384 ->  27476 96.8%  comp 275.7 MB/s  uncomp 1346.5 MB/s
  testdata/cp.html                         :
  LZO:    [b 1M] bytes  24603 ->  11621 47.2%  comp  57.2 MB/s  uncomp 258.5 MB/s
  SNAPPY: [b 4M] bytes  24603 ->  11838 48.1%  comp 190.0 MB/s  uncomp 443.5 MB/s
  testdata/fields.c                        :
  LZO:    [b 1M] bytes  11150 ->   4663 41.8%  comp  73.8 MB/s  uncomp 259.8 MB/s
  SNAPPY: [b 4M] bytes  11150 ->   4728 42.4%  comp 207.3 MB/s  uncomp 431.7 MB/s
  testdata/geo.protodata                   :
  LZO:    [b 1M] bytes 100000 ->  20423 20.4%  comp 115.8 MB/s  uncomp 429.9 MB/s
  SNAPPY: [b 4M] bytes 100000 ->  23488 23.5%  comp 359.0 MB/s  uncomp 625.8 MB/s
  testdata/grammar.lsp                     :
  LZO:    [b 1M] bytes   3721 ->   1781 47.9%  comp  66.8 MB/s  uncomp 311.4 MB/s
  SNAPPY: [b 4M] bytes   3721 ->   1800 48.4%  comp 214.9 MB/s  uncomp 461.2 MB/s
  testdata/house.jpg                       :
  LZO:    [b 1M] bytes 126958 -> 127173 100.2%  comp  20.2 MB/s  uncomp 1420.8 MB/s
  SNAPPY: [b 4M] bytes 126958 -> 126803 99.9%  comp 2037.6 MB/s  uncomp 7578.7 MB/s
  testdata/html                            :
  LZO:    [b 1M] bytes 102400 ->  21027 20.5%  comp 115.1 MB/s  uncomp 423.8 MB/s
  SNAPPY: [b 4M] bytes 102400 ->  24140 23.6%  comp 362.4 MB/s  uncomp 703.5 MB/s
  testdata/html_x_4                        :
  LZO:    [b 1M] bytes 409600 ->  82980 20.3%  comp 120.9 MB/s  uncomp 416.4 MB/s
  SNAPPY: [b 4M] bytes 409600 ->  96472 23.6%  comp 359.0 MB/s  uncomp 698.0 MB/s
  testdata/kennedy.xls                     :
  LZO:    [b 1M] bytes 1029744 -> 357315 34.7%  comp 133.5 MB/s  uncomp 533.6 MB/s
  SNAPPY: [b 4M] bytes 1029744 -> 425735 41.3%  comp 294.1 MB/s  uncomp 432.6 MB/s
  testdata/kppkn.gtb                       :
  LZO:    [b 1M] bytes 184320 ->  71671 38.9%  comp  84.3 MB/s  uncomp 235.9 MB/s
  SNAPPY: [b 4M] bytes 184320 ->  70535 38.3%  comp 231.7 MB/s  uncomp 373.8 MB/s
  testdata/lcet10.txt                      :
  LZO:    [b 1M] bytes 426754 -> 221290 51.9%  comp  57.6 MB/s  uncomp 181.9 MB/s
  SNAPPY: [b 4M] bytes 426754 -> 243710 57.1%  comp 153.7 MB/s  uncomp 341.4 MB/s
  testdata/mapreduce-osdi-1.pdf            :
  LZO:    [b 1M] bytes  94330 ->  76999 81.6%  comp  24.9 MB/s  uncomp 810.9 MB/s
  SNAPPY: [b 4M] bytes  94330 ->  77477 82.1%  comp 709.0 MB/s  uncomp 1669.5 MB/s
  testdata/plrabn12.txt                    :
  LZO:    [b 1M] bytes 481861 -> 294610 61.1%  comp  51.4 MB/s  uncomp 164.8 MB/s
  SNAPPY: [b 4M] bytes 481861 -> 329339 68.3%  comp 129.8 MB/s  uncomp 281.9 MB/s
  testdata/ptt5                            :
  LZO:    [b 1M] bytes 513216 ->  86232 16.8%  comp 119.0 MB/s  uncomp 506.1 MB/s
  SNAPPY: [b 4M] bytes 513216 ->  93455 18.2%  comp 479.8 MB/s  uncomp 670.8 MB/s
  testdata/sum                             :
  LZO:    [b 1M] bytes  38240 ->  17686 46.2%  comp  57.8 MB/s  uncomp 267.0 MB/s
  SNAPPY: [b 4M] bytes  38240 ->  19837 51.9%  comp 194.5 MB/s  uncomp 401.0 MB/s
  testdata/urls.10K                        :
  LZO:    [b 1M] bytes 702087 -> 309320 44.1%  comp  55.7 MB/s  uncomp 265.1 MB/s
  SNAPPY: [b 4M] bytes 702087 -> 357267 50.9%  comp 216.9 MB/s  uncomp 499.3 MB/s
  testdata/xargs.1                         :
  LZO:    [b 1M] bytes   4227 ->   2450 58.0%  comp  56.3 MB/s  uncomp 286.2 MB/s
  SNAPPY: [b 4M] bytes   4227 ->   2509 59.4%  comp 173.8 MB/s  uncomp 397.3 MB/s
This is on a Core2, 64bit, GCC 4.5.2. Please someone port it to C and linux-kernel code so it could be used with zram instead of LZO.