News for hackers. Stuff that matters.
Re:Big Data Lambda Architecture by zacharyvoase on Wednesday January 02, 2013 @09:46AM (#4996898)
So, 'Big Data' has been simplified to 'Problems Solvable by MapReduce'? sigh.

Not every problem can be reduced into a completely cache-able batch job, trivially parallelizable across all of your data. 'Big Data' isn't about breaking up your batch processing into three layers, it's about being smart enough and knowledgeable enough in compsci, statistics, calculus, text processing, regexes, machine learning, business analysis, et cetera, to design an effective system which harvests useful insights from a large bank of atomic, messy, inconsistent data, with an appropriate level of availability and consistency.

The real work is not in using/configuring Hadoop—it's about figuring out what information would bring greater-than-marginal value to a business, and how to compute that efficiently from an existing corpus of data.

There's no silver bullet. Remember?

EDIT: I think the following is particularly disingenuous: "The lambda architecture solves the problem of computing arbitrary functions on arbitrary data in real time by decomposing the problem into three layers"

This is such a ridiculous promise, that it put me in a strongly skeptical mood for the rest of the article.

Re:Big Data Lambda Architecture by jkbyc on Wednesday January 02, 2013 @10:48AM (#4997216)
Agreed. The article looks very weak.

Also, it seems like "delta architecture" would be a better name since the "speed layer" is all about updating views with deltas anyway and the architecture clearly cannot do what it promises.

The view update problem is by far not easy even if you take a subset of a well-defined language like SQL. And the article really seems to assume arbitrary queries, i.e. executing even an arbitrary program on the input data. Just a simple example: "If x is there then remove y" and you have other rules inserting and removing x and y. Then you easily run into a query which given the data does not have a single unambiguous solution and in some cases (loops with an odd number of negations) it is not even clear at all what the solution should be and if it can be defined.

Re:Big Data Lambda Architecture by chubot on Wednesday January 02, 2013 @12:36PM (#4997861)
Regarding the edit, I don't see what's ridiculous about the statement. Can you elaborate?

When he says arbitrary functions, I think he's basically comparing it to the limitations of SQL solutions that are traditionally used for data warehousing. SQL is abused beyond belief and also slow beyond belief for certain types of queries.

MapReduce provides a pretty good place to start from. Sure, it's awkward for certain things. But if you're going to rearchitect a distributed system from scratch for each computation you do, you're not going to make progress very quickly.

See this paper for the opposite viewpoint: http://news.ycombinator.com/item?id=4520057

He provides solutions to common problems which can be expressed with MapReduce.

The lambda architecture is more powerful than what's being advocated in that article (plain mapreduce), although I'm not sure if it's really a thing people are using or just a subject of a book.

Re:Big Data Lambda Architecture by gfodor on Wednesday January 02, 2013 @12:01PM (#4997627)
This is a lot of platitudes but doesn't provide any specific criticisms. Most problems are "Solvable by MapReduce" since map/reduce != Hadoop but is simply an abstraction for computing functions on data in general. The architecture outlined here is a pattern for a generalized system for computing functions on data in real time that has nice properties.
Re:Big Data Lambda Architecture by lvh on Wednesday January 02, 2013 @04:46PM (#4999508)
That it's an abstraction that can do anything isn't sufficient (and I'm not even convinced it's necessary): Turing machines can compute anything too, that doesn't mean they're practical.
Re:Big Data Lambda Architecture by gfodor on Thursday January 03, 2013 @12:00AM (#5001101)
Ok, more platitudes, still waiting for a specific criticism of this architecture and its claimed limitations.

I'll repeat myself: most big data problems fit into the mold of query = function(data), map reduce is a practical substrate for building algorithms to compute these functions, and this paper presents a practical architecture to implement these types of systems.

Re:Big Data Lambda Architecture by lvh on Thursday January 03, 2013 @04:40AM (#5001618)
You made the argument that it was just platitudes and MapReduce is a sufficient model. I was specifically counter-arguing that point; that doesn't mean the claimed model is bad or that I have to reply to every single point in a single comment.