News for hackers. Stuff that matters.
Re:Big Data Lambda Architecture by noelwelsh on Wednesday January 02, 2013 @09:21AM (#4996783)
I'm pretty bullish on the "speed layer" coming to dominate. I've done a fair bit of work with streaming algorithms [1] and they have advantages beyond just latency, reduced memory usage being the primary one. If you believe data is growing faster than computing power it seems that streaming algorithms must be the way forward.

Note that you can do a lot with streaming algorithms (it's not just counting). Also the reduced memory usage (orders of magnitude) makes the complexity of random writes not such a problem as you have less need to go outside a single machine.

[1] Slides on streaming algorithms: http://noelwelsh.com/streaming-algorithms/2012/11/22/streami...

Re:Big Data Lambda Architecture by avibryant on Wednesday January 02, 2013 @11:12AM (#4997342)
Nice slides. You should check out https://github.com/twitter/algebird - we've implemented a number of streaming algorithms (HyperLogLog, Count Min Sketch, along with stuff like minhash and bloom filters) in Scala as Monoid typeclasses. Love to hear your thoughts or get your contributions.
Re:Big Data Lambda Architecture by noelwelsh on Thursday January 03, 2013 @04:42PM (#5004867)
Algebird looks great! Thanks for mentioning it.