Re:Google is investing $3.3B to build clean data centers in ...
by smueller1234 on Friday September 20, 2019 @01:55PM
(#21029445)
You're exactly right. Hadoop and it's ecosystem are... a very poor replica of the system whose predecessor the respective Google papers were written about. There's glaring efficiency things like not supporting complex encodings (last I used HDFS, at least, it could only do full replication) or the query engines on top not implementing sampling in their aggregates as a default/out of the box, which the Google tools do.
Re:Google is investing $3.3B to build clean data centers in ...
by opportune on Friday September 20, 2019 @12:39PM
(#21028600)
Another way to get huge efficiency gains over using Hadoop is to intelligently memoize results and compute as-you-go. But that requires making some bespoke architecture. However I contend that at any scale where it actually makes sense to use Hadoop (especially in the cloud era), it will be cheaper in the medium-term of 1 year or so to dedicate developer time to make something more efficient than a big daily pipeline of Hadoop jobs
Re:Google is investing $3.3B to build clean data centers in ...
by lclarkmichalek on Friday September 20, 2019 @03:14PM
(#21030197)
The good news is that barely anyone at FAANG scale is using Hadoop any more
So here's something you can personally do to make a dent to this problem. Most of the Hadoop jobs that you write will involve some statistical summary over a dataset. Find the total, or the mean, median, 90th quantile, whatever. Writing a Hadoop map-reduce job is the single worst way to do this.Almost always, you can sample say a 1000 points, get a kernel density estimate via Parzen & then use a table. All quantiles, order statistics, functions of order stats...these can be hand computed for several univariate distributions & their location-scale families, and your real-life data can easily be bounded above below by these estimates. So you can get to > 95% accuracy just by hand-calculating. I can go into details if you like, but I suspect most of you already know how to do this.