Re:MapReduce for the Masses: Zero to Hadoop in 5 minutes wit...
by Radim on Friday December 16, 2011 @06:39PM
(#3362816)
I doubt such companies would put their data "in the cloud".
And if they don't, the cost of supporting this "straightforward" Hadoop infrastructure, both in terms of hardware, engineering and support, is so massive that the little elbow grease for a simple I-know-what-it-does solution may well be worth it.
In other words, I share alexro's concerns. If you're buying into the M/R hype and process your blog logs in the cloud, that's one thing. But legitimate business use-cases are probably not as common as people may expect/hope.
Re:MapReduce for the Masses: Zero to Hadoop in 5 minutes wit...
by ajessup on Friday December 16, 2011 @06:48PM
(#3362840)
Hadoop is not a cloud solution - most organizations deploy it on their own infrastructure. Some folks (eg. Amazon AWS) offer a "hosted" version.
Re:MapReduce for the Masses: Zero to Hadoop in 5 minutes wit...
by Radim on Friday December 16, 2011 @09:23PM
(#3363176)
Sure, but that's not what this article is about ("5 min setup, for the masses").
Re:MapReduce for the Masses: Zero to Hadoop in 5 minutes wit...
by nl on Saturday December 17, 2011 @05:18AM
(#3363955)
And if they don't, the cost of supporting this "straightforward" Hadoop infrastructure, both in terms of hardware, engineering and support, is so massive that the little elbow grease for a simple I-know-what-it-does solution may well be worth it.
That's not true. A small cluster and support from a company like Cloudera is much less than an new Oracle install.
Re:MapReduce for the Masses: Zero to Hadoop in 5 minutes wit...
by ericlavigne on Saturday December 17, 2011 @07:52AM
(#3364152)
The mucky mucks can have their answer today with a random subset of that data: 10,000 accounts and the corresponding addresses and transactions. We'll need map-reduce tomorrow if they want precision in the last decimal point.
Re:MapReduce for the Masses: Zero to Hadoop in 5 minutes wit...
by rmc on Monday December 19, 2011 @07:30AM
(#3369426)
You might be able to do it with a random subset, but your accuracy would not be as good. For companies with this much data, 0.1% increase (or decrease) can cost millions. If you use a random subset, you won't be able to accurately find this teeny little variation/
e.g. At a national bank, determine whether distance to the nearest branch or ATM correlates with deposit frequency or average customer relationship value. Your inputs are a) 10 billion timestamped transactions, b) 50 million accounts, c) 200 million addresses and dates for which they began and entered service, and d) a list of 25,000 branch/ATM locations and the dates they entered service.
This is fairly straightforward to describe as a map/reduce job. You could do it on one machine with a few nested loops and some elbow grease, too, but the mucky mucks might want an answer this quarter.
I know the feeling, though: I keep wanting to try it, but haven't been able to find a good excuse in my own business yet.