Re:Rackspace vs. EC2: Performance Analysis
by figured on Monday January 11, 2010 @04:53PM
(#1046122)
Fine print:
"The Bitsource conducted an independent, third-party performance analysis of Rackspace CloudServers and Amazon EC2 on behalf of Rackspace Inc. The results were not influenced in any way by Rackspace Inc. as presented using the methods in this analysis."
Re:Rackspace vs. EC2: Performance Analysis
by allertonm on Monday January 11, 2010 @04:27PM
(#1046060)
Compiling the linux kernel seems like a poor choice of test, since it is not a pure CPU activity and will perform a significant amount of I/O.
In addition, it doesn't seem like compiling the linux kernel requires very much RAM, so you get the most bang for your buck using a CS VM with only 256MB and the cost only goes up from there as you add more RAM that the test cannot benefit from.
(Edit: I would love whoever downmodded me to explain why I am wrong.)
Re:Rackspace vs. EC2: Performance Analysis
by pquerna on Monday January 11, 2010 @04:35PM
(#1046087)
its a decent enough test, mostly because most geeks out there have done it themselves, so you can get a good 'feel' for a machine by knowing how long it takes to compile the kernel.
its not perfect, and yeah, its kinda a geeky-benchmark, but it doesn't make it an invalid benchmark, as long as it was conducted correctly.
Re:Rackspace vs. EC2: Performance Analysis
by allertonm on Monday January 11, 2010 @04:45PM
(#1046105)
I consider myself a geek (and most of my friends would agree) but it's been 15 years since I last compiled the Linux kernel :)
That said, I wouldn't rule it out as a test because of that, but I do think that if you are going to compare 4 different RAM sizes of CS instances, and compare the costs of doing a job using those different sizes then you need to pick a job that is going to make use of the RAM. You can see from the results that there is a small (time) benefit moving from 256 to 512MB (but not enough to offset the increased costs) and virtually none above that.
Re:Rackspace vs. EC2: Performance Analysis
by ajross on Monday January 11, 2010 @04:04PM
(#1045992)
Crank curmudgeon review: Loses big points for including illegibly down-filtered graphs in the page. Either make them readable or make them links. Using fancy animations isn't an excuse for including 600x266 pixel unreadable blotches in your document.
Re:Rackspace vs. EC2: Performance Analysis
by SamAtt on Monday January 11, 2010 @04:10PM
(#1046005)
All the graphs have links to full size versions and the size in the article itself is dictated by the site (since the side bar is clearly mandatory) So I don't quite no what your problem is.
Re:Rackspace vs. EC2: Performance Analysis
by ajross on Monday January 11, 2010 @04:19PM
(#1046040)
I can't read them. What's the point to putting them there if I can't read them? Just include a link saying "Graph...". Why format a graph to be 1200 pixels wide, with small fonts even at that size, and then shrink it by a factor of two?
I mean, the inline images might as well be pictures of puppies for all the value they provide to the page.
Edit: I should add that if the size of the images is dictated by the site, then some attention might have been provided to including inline images designed to be used at that size. It's not like it's impossible to display this information in an 600-pixel-wide format...
Re:Rackspace vs. EC2: Performance Analysis
by moe on Monday January 11, 2010 @07:37PM
(#1046460)
Amen.
Rackspace knows that very well, wonder why they didn't tell the author to use EBS for the comparison. Oh wait, I might have an idea...
Anyways, as usual, don't trust any benchmark that you haven't faked yourself. This one, just like all other rackspace/ec2 comparisons that you see cropping up these days, was paid for by Rackspace and obviously tweaked to produce their desired outcome.
Ironically they even point out the flaw in their setup by showing how the CPUs on the EC2 instances are not fully utilized.
Guess what that means? It means the test was I/O bound.
I'd be curious to see the test repeated, but this time with a RAID10 over a couple EBS volumes. Not sure Rackspace will pay for that one, though...
Re:Rackspace vs. EC2: Performance Analysis
by dgreensp on Monday January 11, 2010 @07:58PM
(#1046518)
In my own informal experience, running identical CPU-bound tasks on Rackspace and EC2, it's just a lot easier to get good performance on Rackspace.
Instance size on Rackspace does affect performance (as their FAQ explains), but it affects minimum guaranteed performance; there's bursting up to the capacity of the entire machine you have a slice of. I know there's something unsettling about relying on excess capacity, but in practice any Rackspace instance will tend to beat any but the most expensive EC2 instance in terms of raw CPU.
The sweet spot of EC2 really is offline processing where hundreds or thousands of machines are spun up by software. If you want to manually fire up a small but flexible number of fast Linux machines, I would recommend Rackspace.
The analysis may be "independent", but the fine print at the bottom of the article indicates that it was paid for by Rackspace.
Yea. Right.
Rackspace just loves to pay for random pseudo-benchmarks. It's their way to give something back to the community. Or something.
In addition, it doesn't seem like compiling the linux kernel requires very much RAM, so you get the most bang for your buck using a CS VM with only 256MB and the cost only goes up from there as you add more RAM that the test cannot benefit from.
(Edit: I would love whoever downmodded me to explain why I am wrong.)
its not perfect, and yeah, its kinda a geeky-benchmark, but it doesn't make it an invalid benchmark, as long as it was conducted correctly.
That said, I wouldn't rule it out as a test because of that, but I do think that if you are going to compare 4 different RAM sizes of CS instances, and compare the costs of doing a job using those different sizes then you need to pick a job that is going to make use of the RAM. You can see from the results that there is a small (time) benefit moving from 256 to 512MB (but not enough to offset the increased costs) and virtually none above that.
I mean, the inline images might as well be pictures of puppies for all the value they provide to the page.
Edit: I should add that if the size of the images is dictated by the site, then some attention might have been provided to including inline images designed to be used at that size. It's not like it's impossible to display this information in an 600-pixel-wide format...
Rackspace knows that very well, wonder why they didn't tell the author to use EBS for the comparison. Oh wait, I might have an idea...
Anyways, as usual, don't trust any benchmark that you haven't faked yourself. This one, just like all other rackspace/ec2 comparisons that you see cropping up these days, was paid for by Rackspace and obviously tweaked to produce their desired outcome.
Ironically they even point out the flaw in their setup by showing how the CPUs on the EC2 instances are not fully utilized.
Guess what that means? It means the test was I/O bound.
I'd be curious to see the test repeated, but this time with a RAID10 over a couple EBS volumes. Not sure Rackspace will pay for that one, though...
Instance size on Rackspace does affect performance (as their FAQ explains), but it affects minimum guaranteed performance; there's bursting up to the capacity of the entire machine you have a slice of. I know there's something unsettling about relying on excess capacity, but in practice any Rackspace instance will tend to beat any but the most expensive EC2 instance in terms of raw CPU.
The sweet spot of EC2 really is offline processing where hundreds or thousands of machines are spun up by software. If you want to manually fire up a small but flexible number of fast Linux machines, I would recommend Rackspace.
Do they feel their cloud offerings are inferior to Amazon's?
It's also nice to see the tests being done over a period of time, rather than a single day.