News for hackers. Stuff that matters.
Re:The big data disaster by PaulHoule on Thursday November 22, 2012 @09:49AM (#4818913)
huh?

for me "big data" is the emergence of scalable tools for handling data sets that are much bigger than you can handle with one computer. (ex. Hadoop)

there's definitely a feeding frenzy here because the spend on hardware could get high (say 200 powerful servers), therefore the spend on associated software and services could be high too.

a few issues stalk big data

for any IT project there's the issue of how much value you can create per bit. Imagine Facebook has a $10 ARPU, for instance. They can't afford to use more than $10 a year worth of hardware and electricity to offer you the service.

Thus, if your "big data" project is going to cost $500,000 you ought to have a plan to create more than that in value.

often the best way to deal with "big" data is to make it small. I had my eyes on a 50 TB data set a few months ago but didn't have the budget to work with the whole thing.

I realized I could take a statistical sample of just 50 GB which I could download to my house and process on my cluster here. I can't get an accurate answer to ~every~ possible question with my 50GB sample, but I got answers to the most pressing questions and discovered that a 1 TB sample would be good enough for the toughest questions I wanted to ask.

Re:The big data disaster by Anon84 on Thursday November 22, 2012 @10:05AM (#4818955)
I'm curious what those 50TB were of. I don't know of many datasets of that size.