These benchmarks were done by someone who seems to have no idea about the ins-and-outs of EC2. For example, there's a lot of IO bound tests... but they're on instance storage. What about Instance storage vs EBS? What about network performance over time?

All I can get from these is that Amazon isn't lying that their instance sizes are different in terms of compute units and memory.

It looks strange to me that High-CPU Instances didn't come as clear winners in compilation tests. I thought of choosing only between c1.medium and c1.xlarge for my CPU-intensive tasks. Looks like I should look closely into all types of instances instead, except for micro and small.

According to the specs Amazon gives us, "High-Memory" instances have 3.25 Compute Units per virtual core, while "High-CPU" only 2.5. It would be interesting to know what is going on: are their compilation tests unable to properly utilize many cores, or the nodes assigned to "High-Memory" instances tend to have more powerful hardware, or what?

How about a comparison to a known standalone unvirtualized box?

I was hoping for more extensive and definitive. Bandwidth? Latency? Startup time? A meaningful sample with variation from day to day and an idea of variation across instances running at the same time?

For all the dollars being thrown at EC2, I'm surprised someone hasn't gone nuts with benchmarking this for the authority, notoriety and seo link juice.

I've done some long-running benchmarks of EC2, mostly for kicks and personal edification. For about 8 months now, I've had a job running on an EC2 "small" instance, that performs a suite of microbencharks every 10 seconds and records the results. Lots of raw data can be found here:

http://amistrongeryet.com/dashboard.jsp

This covers a variety of AWS operations, ranging from simple EC2 CPU benchmarks to SimpleDB and RDS operations. I also include some measurements of Google App Engine. I've put a lot of focus on measuring variation over time, and measuring outliers (i.e. 99.9th percentile, not just mean). I'd like to measure variation across instances, but haven't gotten around to it.

The web site linked above has no documentation at the moment, and is a bit broken in other ways, but the data may be of interest. I have a little blog (http://amistrongeryet.blogspot.com/) where I've discussed some of my findings, e.g. http://amistrongeryet.blogspot.com/2010/04/early-results-on-..., http://amistrongeryet.blogspot.com/2010/04/beginnings-of-rob..., and http://amistrongeryet.blogspot.com/2010/04/cloud-data-storag....

If there's interest, I'll put a little effort into cleaning up and documenting the data reporting site.

Hi, I'm definitely interested in benchmarking the EC2 instances. Are you doing some benchmarks that are CPU-intensive? You're measuring the one instance that you've got to use as a server, aren't you? (I'm interested more in, "what perfomance would I keep getting if I keep requesting instances") How can I contact you?

You can reach me at e-mail address "public" on domain "snewman.net". I'd be happy to share more of what I'm doing.

http://cloudharmony.com/ has at least as extensive benchmarks, and includes other cloud providers.

I wonder how those compare to others IaaS providers like Gogrid or Rackspace

How about cc1.4xlarge we have been debating using them for the IO.

i'm much more interested in ec2's variation over time