News for hackers. Stuff that matters.
Re:Google is investing $3.3B to build clean data centers in ... by dxbydt on Friday September 20, 2019 @12:24PM (#21028415)
I did a bunch of work on managing/anticipating the power draw. ( https://news.ycombinator.com/item?id=13747481 ). Something nobody talks about is how grossly wasteful & inefficient Hadoop is. There's a bunch of cited papers (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.58... ) that study power consumption of Hadoop nodes, but nothing on how to actually reduce the consumption.

So here's something you can personally do to make a dent to this problem. Most of the Hadoop jobs that you write will involve some statistical summary over a dataset. Find the total, or the mean, median, 90th quantile, whatever. Writing a Hadoop map-reduce job is the single worst way to do this.Almost always, you can sample say a 1000 points, get a kernel density estimate via Parzen & then use a table. All quantiles, order statistics, functions of order stats...these can be hand computed for several univariate distributions & their location-scale families, and your real-life data can easily be bounded above below by these estimates. So you can get to > 95% accuracy just by hand-calculating. I can go into details if you like, but I suspect most of you already know how to do this.

Re:Google is investing $3.3B to build clean data centers in ... by smueller1234 on Friday September 20, 2019 @01:55PM (#21029445)
You're exactly right. Hadoop and it's ecosystem are... a very poor replica of the system whose predecessor the respective Google papers were written about. There's glaring efficiency things like not supporting complex encodings (last I used HDFS, at least, it could only do full replication) or the query engines on top not implementing sampling in their aggregates as a default/out of the box, which the Google tools do.
Re:Google is investing $3.3B to build clean data centers in ... by opportune on Friday September 20, 2019 @12:39PM (#21028600)
Another way to get huge efficiency gains over using Hadoop is to intelligently memoize results and compute as-you-go. But that requires making some bespoke architecture. However I contend that at any scale where it actually makes sense to use Hadoop (especially in the cloud era), it will be cheaper in the medium-term of 1 year or so to dedicate developer time to make something more efficient than a big daily pipeline of Hadoop jobs
Re:Google is investing $3.3B to build clean data centers in ... by lclarkmichalek on Friday September 20, 2019 @03:14PM (#21030197)
The good news is that barely anyone at FAANG scale is using Hadoop any more