Re:Google's Secret Weapon: MapReduce
by jgrahamc on Friday December 12, 2008 @05:43AM
(#395568)
It's interesting how much noise there is around MapReduce. I'm working as CTO of a start-up with a very large dataset (terabytes) where we need to do quite complex queries across the dataset and very quickly.
Naturally we are taking a distributed approach to the problem since hardware is cheap and relatively easy to coordinate via software.
We decided to go our own way using a mixture of some databases and raw file system access with our own application coordinating both the cluster of machines and performing the queries themselves after performing extensive benchmarking of everything.
We looked closely at Hadoop and came to the conclusion that MapReduce was just the wrong architecture for our problem. Its performance was horrible.
The moral of this story is that MapReduce is interesting, but that doesn't prevent you from having to actually test algorithms and make an informed choice. Just because Google's using it doesn't mean it's a panacea.
Naturally we are taking a distributed approach to the problem since hardware is cheap and relatively easy to coordinate via software.
We decided to go our own way using a mixture of some databases and raw file system access with our own application coordinating both the cluster of machines and performing the queries themselves after performing extensive benchmarking of everything.
We looked closely at Hadoop and came to the conclusion that MapReduce was just the wrong architecture for our problem. Its performance was horrible.
The moral of this story is that MapReduce is interesting, but that doesn't prevent you from having to actually test algorithms and make an informed choice. Just because Google's using it doesn't mean it's a panacea.