We've been facing that question in our HPC lab. We dropped off the top500 for the first time in 5+ years. We just installed a new cluster, but during our scheduled outage of 3 days we were unable to get a good benchmark with linpack. We've had some interesting throttling issues on our new cluster. Once we started figuring it out, we wanted to benchmark it and submit it in time for the november list. We couldn't justify kicking everyone off, but in the event that we had a catastrophic event, we had it set to run a benchmark before coming back online. Nothing happened (thankfully.. and regretably) so we were not able to submit a benchmark for our new 6000 core cluster.

Any chance you guys ran it on the new GPU cluster before opening it to the public? Would have been cool to get the benchmark on that entire thing b/f customers started running stuff on it.

Not that I am aware of, but the AWS team is big (and getting bigger) and lots of stuff happens that I know nothing about.

Which naturally begs the question, "would you?"

I can't see any circumstance under which we would chase away paying customers. Benchmarks are awesome, but keeping customers happy is better than awesome.