News for hackers. Stuff that matters.
Re:Google releases snappy, the compression library used in B... by tptacek on Tuesday March 22, 2011 @07:04PM (#2356967)
When you can measure efficiency improvements like this in millions of dollars, I'm sure this makes a whole hell of a lot of sense. But for anyone below, say, Twitter's scale: is this ever an engineering win over zlib?
Re:Google releases snappy, the compression library used in B... by btilly on Tuesday March 22, 2011 @09:26PM (#2357468)
Many years ago I had to do a bunch of sorting and merging datasets on some underpowered hardware. I found that I got significant speedups by compressing my data during mergesorts because I was able to squeeze extra passes in RAM before data had to hit disk. (To be clear, I was writing to and from files, but found the data still in cache when I went to read it again.) If compression/decompression were an order of magnitude faster, the win could have been even bigger and more obvious.

Don't think of this as, "We can cheaply compress large amounts of data to save space." Think of this as, "We can compress/decompress stuff on the fly just because it is convenient for us." For the latter kind of usage the efficiency is an enabler, and the fact that the compression is supposed to be temporary makes interacting with other things a complete non-issue.

Re:Google releases snappy, the compression library used in B... by lemming on Tuesday March 22, 2011 @07:22PM (#2357050)
It's a little difficult to say. One space where there seems to be very little research that I've been investigating recently is compression for database records (or in our case, serialised complex objects stored in a data grid) where the statistical model is build across many records and then is constant during compression and decompression, as opposed to being adaptive. This means that you can exploit global redundancy across many records and you don't need to store the model with every item, which is good if your items are small, you store zillions of them and space is at a premium (i.e. you're storing it in RAM).

These use cases are normally pretty application specific though, so I imagine a lot of in-house code gets written for things like this. Seems like a pretty similar use case to this, I'd be interested in seeing details of their algorithm - I can't see it anywhere obvious on their site.

Re:Google releases snappy, the compression library used in B... by alecco on Tuesday March 22, 2011 @08:40PM (#2357315)
There's a lot to DB compression depending on the type of data access. The PFOR approach is amazing and simple for data arranged in columns without much deviation. For serialized trees you usually end up in LZ* land. But with careful organization it can be tolerable.
Re:Google releases snappy, the compression library used in B... by Andys on Tuesday March 22, 2011 @07:08PM (#2356993)
Perhaps a more pertinent question: Is it any better than LZJB or LZO?
Re:Google releases snappy, the compression library used in B... by alecco on Tuesday March 22, 2011 @08:40PM (#2357318)
The LZO codebase is one of the worse spaghetti code messes I've seen in my life.
Re:Google releases snappy, the compression library used in B... by wmf on Tuesday March 22, 2011 @07:16PM (#2357025)
It's got a better license than LZJB (and maybe LZO depending on your views).
Re:Google releases snappy, the compression library used in B... by ot on Tuesday March 22, 2011 @07:25PM (#2357065)
Doesn't Google use GPL2 all over the place?
Re:Google releases snappy, the compression library used in B... by wmf on Tuesday March 22, 2011 @07:33PM (#2357097)
Just to use one example, I would assume that the crawler/indexer/ranker "secret sauce" in the appliance can't link against any GPL libraries.
Re:Google releases snappy, the compression library used in B... by tptacek on Tuesday March 22, 2011 @07:57PM (#2357163)
If you don't distribute your GPL-tainted code, I don't believe you need to do anything to comply with the GPL. This is why the Affero GPL exists.
Re:Google releases snappy, the compression library used in B... by tonfa on Tuesday March 22, 2011 @08:04PM (#2357188)
The keyword in the parent comment was "appliance" :)
Re:Google releases snappy, the compression library used in B... by tptacek on Tuesday March 22, 2011 @08:10PM (#2357210)
Doh! Missed that. Yep.
Re:Google releases snappy, the compression library used in B... by lemming on Tuesday March 22, 2011 @07:22PM (#2357054)
And LZO.
Re:Google releases snappy, the compression library used in B... by alecco on Tuesday March 22, 2011 @08:22PM (#2357254)
Websites will benefit indirectly from this through packages getting better performance (like databases or caching systems). If this algorithm is as good as Google reported a few years ago this is big, big news.

Note: have you seen the LZO code? I bet not.