Yes exactly, the speed of throwing up pre-built AMIs is tons faster than building from scratch each time you launch.

However, we're probably going to make a deployment script (we use fabric) that builds the AMI from scratch. Then, when we need to update the AMI, we just update the fabric script and run it to make the new AMI. That way if we ever need to make AMI's in a different region, we can just run the fabric script in that region.

I use a similar process to this in my infrastructure - I use Puppet to configure instances, then a Fabric command to create new AMIs based on these Puppet configurations.

This gives the best of both worlds - version controlled configuration files, and an automated process for making new AMIs. Instances also boot quickly, as nearly all of the configuration is baked-in to the image.

Using EC2 tags gives you even more room for automation here.

Can you elaborate on how you use EC2 tags? The only thin we use them for is to mark a new instance as bootstrapped once puppet finishes.

It also sounds like you could pretty easily make your AMI creation step a job in your CI software.

If you used puppet (or chef etc) instead of fabric for config, you could have the best of both worlds - it should be much faster to run puppet on existing instances to keep them up to date, rather than failover all your instances to new AMIs every time. But by still creating those AMIs, you'll have them ready when you need to launch additional instances.

The correct way to do it is as follows.

1) Set up a test environment using puppet or whatever that boots up an image instance and deploys your whole stack to your cloud provider. 2) Deploy your release candidate to this environment. 3) Run some smoke tests, acceptance tests and maybe even a few performance tests against this environment. 4) If all the tests pass store your image somewhere (S3, GitHub) also make sure to tag the image with your release candidate version. 5) You can now deploy this change-set to your production environment.

When things go wrong you will have fully configured servers available almost immediately.

Now granted this is potentially a very long running process but you could simply set this up as a nightly build.

That's interesting, but would not work for us since we ship code up to 20-30 times a day at Canvas (using continuous deployment). It's already bad enough that it takes about 10 minutes to ship.

If you really want to make it funked out, you could set up a CI process to check out your puppet scripts and build an AMI whenever a change is made to your config. Then the deploy task would be a rolling upgrade of the new AMI to whichever environment you want the config to affect. With some cloud formation and puppet trickery you could also one-click-bootstrap new environments (e.g. "production in region x", or "staging in region y") and just run the deploy task to that new environment.

Since you're already using puppet, which is great at keeping existing instances up to date with config changes (it's almost the whole point), wouldn't it be better for your CI to create those AMIs asynchronously when you deploy, but not actually use them unless you (or autoscale) are launching new instances? Just use puppet to deploy config changes as usual, then have the AMIs at the ready, rather than immediately recycling your instances with the new AMIs. This should be much faster (and more loosely decoupled from AWS).