I have not always had the luxury of running on EC2. When you run on your own hardware this is not a given.
Eh? VMs.
If you run at near-max load:
If nothing else you should have a staging environment and that can be your second cluster too.
We are talking about Heroku here, they are more expensive than EC2 so you can get away with this on similar budgets too. ;)
Especially here: > Puppet? I am sorry but Puppet, chef and their ilk are horrible, who knows what state any machine is ever in at any point in time. humans will inevitably forget to ensure absent, or make an assumption about state in puppet land.
You setup the new cluster, switch the load balancer, tear down the old cluster.
I suggest you look into blog posts like this [if this is a pain point for you]: http://blog.codeship.io/2013/09/12/the-codeship-workflow-par...