News for hackers. Stuff that matters.
Re:Improving MapReduce Performance by strlen on Thursday December 17, 2009 @08:27PM (#1002354)
Great article Todd,

One key thing to highlight is the importance of compression and using a streaming compression algorithm. Compression means there's less data to transfer (across the network and -- even more importantly -- from disk), which means the transfers will complete faster.

Not only does it allow your compressed files to be splittable (not possible with a conventional compression algorithm which requires all compressed data to have its own Huffman tree), it runs very quickly and easily adopts to a _stream_ (rather than a monolithic chunk) of data.

We've just added support for LZF (a similar arithmetic/streaming compression codec) into Voldemort and performance results have been great:

http://groups.google.com/group/project-voldemort/browse_thre...

Here's some background: http://en.wikipedia.org/wiki/Arithmetic_coding http://en.wikipedia.org/wiki/Lempel_Ziv

(I had the good fortune to take an information theory class during undergrad)

There are no comments on this comment yet.