One of the more common scenarios where CF gets into a broken state is:
1) create new S3 bucket + something else (e.g. some Elastic Beanstalk env update)
2) something else fails, causing rollback
3) S3 bucket already contained data (e.g. your failed Elastic Beanstalk env update caused it to write data)
4) CF refuses to destroy the S3 bucket, entering a "rollback failed" state
In this cause, manually wiping the S3 bucket works well enough. But generally, it appears that CF works kind of when the updates you're making are really small, incremental updates.
Sometimes it gets totally corrupted and you need to nuke stuff, per your advice. This automatically leads me to the following suggestion: leave mission-critical data out of CloudFormation. Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.
Check out the docs here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
This a major pitfall when using DeletionPolicy=Retain with named resources. It breaks seamless rollbacks/rollforwards. If you rollback, in order to deploy again you need to either delete the named resources with DeletionPolicy=retain that were rolled back, or update your template to rename them all. It is such a huge pain.
A nice feature would be a "ForceDelete" deletionpolicy where it would delete the objects. You can even set this initially when creating a stack, and change it to "Retain" later when the stack is stable.
Totally agree btw that it's a huge pain initially, though once you know it it's also not that hard to work around.
There are ways around it as stated below, but I agree completely. I don’t bother with CloudFormation with cross shared, building block type infrastructure like RDS and ElasticSearch. It’s just not that much of a pain to spin up a database on each account. Besides, the characteristics of the databases in different environments are going to be so different, that you are going to either have parameters or FindInMap functions anyway so for all intents and purposes, you’re not running the same template anyway.
As the article said, changing any resource that’s exported from a CF template is such a pain, it would be better just to use parameter store if you can get away with it.
Try to avoid naming resources ("Name" property), so there would be no clashes. Use Ref (same stack) or ImportValue (other stack) to reference created resources. If you want, you may concatenate (Fn:Join "AWS:StackName", "-public-elb"). Cloud way is you replace things, rather than keeping it. It is convenient to know, that if I delete CF stack, everything is cleaned.
Speaking of S3, its better not to include bucket resource in CF until you know what you're doing.
> stuff like RDS databases which you absolutely never ever want to have destroyed
That begs for a separate CF stack, with template creating RDS and related things only, then export endpoint in Outputs.
The problem introduced by that approach is how to manage a large number of CF stacks. First I used a homegrown Python library to manage them, then switched to having Terraform manage CF.
At first Terraform on CF was just intended to be an expedient measure to facilitate migrating everything to Terraform, and eventually we did migrate to pure Terraform. But then we started hitting all the rough edges in Terraform. In hindsight, the hybrid approach had actually been more stable and manageable than using either tool in isolation.
Terraform also lacks a lot of the extra "smarts" that CF has; like rolling updates of any kind and some other higher level automation across different services. They take very much a blue/green approach which is beyond limiting for some services.
Ran into a few broken states using Cloud Formation via Serverless project. Luckily had been in the habbit of keeping "statefull" AWS resources such as queues, databases, and other stuff in their own stacks(or Terraform) to keep their stacks complication to a minimum and mitigate the impact of the more complicated stacks needing to be deleted..
Ultimately I started going with and promote a hybrid approach where Terraform makes sense; Terraform for base infrastructure(including most stateful resources) and CF for stuff like autoscale groups and etc.
My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild it. Even if it is fixable on paper, being able to just nuke a stack and replace it is a very good thing. This has happened to us multiple times and having a plan helps.
So what I do with elasticsearch for example is use 3 CF stacks (one for each AZ). This allows me to do things like rolling restarts in a sane way without having to do some flaky deep integration into CF to make it orchestrate a rolling restart without destroying my cluster state simply by replacing the stacks one by one.
If I were to build this again, I'd probably use terraform. Also, I'm looking forward to moving most of our stuff to kubernetes.