Re:Most data isn’t “big,” and businesses are wasting money p...
by scott_w on Monday May 13, 2013 @04:44AM
(#5697738)
My feeling is that the title of the article rails against companies who think they need Big Data when they don't.
Re:Most data isn’t “big,” and businesses are wasting money p...
by walshemj on Monday May 13, 2013 @06:02AM
(#5697933)
even analysing say a small number of serp results for say 40k keywords per month quickly generates lots of data.
And If the proposal to analyse a 1/4 million terms for kelly search had gone live I woudl have been creating over a LOC (Libray of congress) for each run.
Nonetheless, large jobs are important too. Over 80% of the IO and over 90% of cluster cycles are consumed by less than 10% of the largest jobs (7% of the largest jobs in the Facebook cluster). These large jobs, in the clusters we considered, are typically revenue-generating critical production jobs feeding front-end applications.
So MR job characteristics might follow a power law distribution, and @mims is focusing on one end of the tail. Sure, that's cool!
But then @mims also selectively quotes the TC article, which ends with an excellent point that contradicts his thesis:
The big data fallacy may sound disappointing, but it is actually a strong argument for why we need even bigger data. Because the amount of valuable insights we can derive from big data is so very tiny, we need to collect even more data and use more powerful analytics to increase our chance of finding them.
I think @mims over-pursues the stupid Forbes/BI straw man here. As one would expect with data, the story is complicated. Mom and pop stores don't need to worry about Cloudera's latest offering, but companies working on the cutting edge of analysis still absolutely need tools like Hadoop, Impala, and Redshift.