Re:The big data disaster
by PaulHoule on Thursday November 22, 2012 @09:49AM
(#4818913)
huh?
for me "big data" is the emergence of scalable tools for handling data sets that are much bigger than you can handle with one computer. (ex. Hadoop)
there's definitely a feeding frenzy here because the spend on hardware could get high (say 200 powerful servers), therefore the spend on associated software and services could be high too.
a few issues stalk big data
for any IT project there's the issue of how much value you can create per bit. Imagine Facebook has a $10 ARPU, for instance. They can't afford to use more than $10 a year worth of hardware and electricity to offer you the service.
Thus, if your "big data" project is going to cost $500,000 you ought to have a plan to create more than that in value.
often the best way to deal with "big" data is to make it small. I had my eyes on a 50 TB data set a few months ago but didn't have the budget to work with the whole thing.
I realized I could take a statistical sample of just 50 GB which I could download to my house and process on my cluster here. I can't get an accurate answer to ~every~ possible question with my 50GB sample, but I got answers to the most pressing questions and discovered that a 1 TB sample would be good enough for the toughest questions I wanted to ask.
for me "big data" is the emergence of scalable tools for handling data sets that are much bigger than you can handle with one computer. (ex. Hadoop)
there's definitely a feeding frenzy here because the spend on hardware could get high (say 200 powerful servers), therefore the spend on associated software and services could be high too.
a few issues stalk big data
for any IT project there's the issue of how much value you can create per bit. Imagine Facebook has a $10 ARPU, for instance. They can't afford to use more than $10 a year worth of hardware and electricity to offer you the service.
Thus, if your "big data" project is going to cost $500,000 you ought to have a plan to create more than that in value.
often the best way to deal with "big" data is to make it small. I had my eyes on a 50 TB data set a few months ago but didn't have the budget to work with the whole thing.
I realized I could take a statistical sample of just 50 GB which I could download to my house and process on my cluster here. I can't get an accurate answer to ~every~ possible question with my 50GB sample, but I got answers to the most pressing questions and discovered that a 1 TB sample would be good enough for the toughest questions I wanted to ask.