News for hackers. Stuff that matters.
Re:Sorting Petabytes with MapReduce by abtinf on Thursday September 08, 2011 @12:00AM (#2972582)
Google keeps bragging about all their internal capabilities because they want to hire people. But so what? If I had a real reason to, I could fire up a couple thousand machines on amazon and analyze data akimbo.

In a sense, google is worse than microsoft - they really don't share any of their hardcore cs innovations. At least MS is in the business of selling technology. Google is in the business of hoarding it in order to derive competitive advantage in advertising.

I just wrote up an entry about this at http://news.ycombinator.com/item?id=2972368

Re:Sorting Petabytes with MapReduce by someone13 on Thursday September 08, 2011 @12:12AM (#2972604)
I disagree with parts of this - Google does release some rather high-quality source code (LevelDB, Protocol Buffers, Chrome/V8), and also provides some well-done research papers.

The entire concept of MapReduce wouldn't exist without Google. And that's inspired a whole host of products. In the same vein, I don't find it unusual that they're NOT sharing the code for MapReduce et al., in that it's proprietary technology. I don't expect Amazon to share the source code behind AWS, and I don't expect Google to share the source code for MapReduce.

Re:Sorting Petabytes with MapReduce by abtinf on Thursday September 08, 2011 @12:15AM (#2972608)
Your points are totally valid.

But my point of view is, which company is more relevant to me as a developer and as someone trying to build a startup? And it seems to me that I get a lot more love from Amazon, whether its some nifty new DNS feature or dropping incoming bandwidth charges to 0.

Re:Sorting Petabytes with MapReduce by DasIch on Thursday September 08, 2011 @12:58AM (#2972693)
That doesn't change the fact that unlike Amazon, Google is not in the business of selling tools to developers; with that in mind it is impressive what they do for developers despite that.
Re:Sorting Petabytes with MapReduce by abraham on Thursday September 08, 2011 @02:30AM (#2972850)
I don't see that as being a very good comparison. Amazon provides a great service but you have to pay Amazon to get access. Google provides a lot of open source code that you can take, modify, and run on any hardware.
Re:Sorting Petabytes with MapReduce by moultano on Wednesday September 07, 2011 @09:23PM (#2972226)
One of my favorite parts of my job is the ability to grab thousands of machines at low priority for the hell of it.
Re:Sorting Petabytes with MapReduce by packetslave on Wednesday September 07, 2011 @10:56PM (#2972456)
...and without asking permission. "Gee, I wonder how much faster this would be on 5000 machines instead of 2000?" clickicky-click ... "yep, quite a bit faster."
Re:Sorting Petabytes with MapReduce by jsnell on Thursday September 08, 2011 @09:50AM (#2973880)
Not permission, but occasionally forgiveness. Task isolation isn't perfect, and backends don't scale infinitely.
Re:Sorting Petabytes with MapReduce by boyter on Wednesday September 07, 2011 @11:21PM (#2972505)
As someone at a large corp who just had a 3 day battle to get a new VM with 2 gig of ram and a shared CPU I suddenly feel like crying.

I guess I have no real use for something of this magnitude, but I would still like to get my hands on a seriously large cluster just for the sake of expanding my experience.

I don't suppose anyone knows of a cost effective way of doing so? I have spun up my own Hadoop clusters before but with only 4 machines its not exactly exciting.

Re:Sorting Petabytes with MapReduce by jonburs on Thursday September 08, 2011 @12:31AM (#2972638)
Amazon -- you can use Elastic Map Reduce [1] (the easiest way to run a transient Hadoop cluster) or spin up a bunch of EC2 nodes and deploy your own stack. Both methods can provide you access to very powerful hardware [2].

(Disclaimer: I work for Amazon, but not in AWS, and these are my own personal opinions.)

[1] http://aws.amazon.com/elasticmapreduce/

[2] http://aws.amazon.com/ec2/instance-types/

Re:Sorting Petabytes with MapReduce by epistasis on Thursday September 08, 2011 @04:24AM (#2973100)
I don't work for Amazon, have hundreds of non-AWS CPUs at my disposal, and still prefer AWS, as long as you don't have to do much I/O.

Once there's a few terabytes of data or more, it's best to be off of AWS, because you'll never be able to saturate your CPUs. Even for smaller datasets of 10s of gigabytes, it's kind of iffy.

Re:Sorting Petabytes with MapReduce by brandonb on Thursday September 08, 2011 @01:10AM (#2972713)
Their benchmarks seem a little odd. They claim an 11x speed improvement, compared to using half as many computers three years ago. But you'd expect roughly an order of magnitude improvement anyway, just due to Moore's law.

I'd be really curious to see what part of the speedup is due to software optimizations, i.e., compare the 2011 software with the 2008 software on identical hardware.

Re:Sorting Petabytes with MapReduce by tonfa on Thursday September 08, 2011 @02:47AM (#2972889)
From the article: "a large part of the credit goes to numerous advances in Google's hardware, cluster management system, and storage stack".
Re:Sorting Petabytes with MapReduce by mda on Thursday September 08, 2011 @02:50AM (#2972901)
Moore's law does not suggest order of magnitude performance increase in 3 years.
Re:Sorting Petabytes with MapReduce by brandonb on Thursday September 08, 2011 @05:04AM (#2973173)
Moore's law suggests 4x performance increase over 3 years, and they doubled the number of machines, so you end up with roughly an 8x performance improvement from "just" hardware.
Re:Sorting Petabytes with MapReduce by mda on Friday September 09, 2011 @05:34PM (#2979955)
No, Moore's law suggests 4x increase in transistor count over 3 years.