News for hackers. Stuff that matters.
Re:Two years with CloudFormation: lessons learned by jillesvangurp on Sunday August 12, 2018 @09:35AM (#17744291)
We've been using CF for a few years as well. IMHO it is very complicated to manage and getting stuff working involves a lot of trial and error. Also you end up waiting a lot. Waiting for things to spin up, waiting for things to become available, waiting for things to rollback, etc. On top of that the failure modes can be ugly and hard to figure out.

My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild it. Even if it is fixable on paper, being able to just nuke a stack and replace it is a very good thing. This has happened to us multiple times and having a plan helps.

So what I do with elasticsearch for example is use 3 CF stacks (one for each AZ). This allows me to do things like rolling restarts in a sane way without having to do some flaky deep integration into CF to make it orchestrate a rolling restart without destroying my cluster state simply by replacing the stacks one by one.

If I were to build this again, I'd probably use terraform. Also, I'm looking forward to moving most of our stuff to kubernetes.

Re:Two years with CloudFormation: lessons learned by stingraycharles on Sunday August 12, 2018 @10:04AM (#17744393)
> My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild

One of the more common scenarios where CF gets into a broken state is:

1) create new S3 bucket + something else (e.g. some Elastic Beanstalk env update)

2) something else fails, causing rollback

3) S3 bucket already contained data (e.g. your failed Elastic Beanstalk env update caused it to write data)

4) CF refuses to destroy the S3 bucket, entering a "rollback failed" state

In this cause, manually wiping the S3 bucket works well enough. But generally, it appears that CF works kind of when the updates you're making are really small, incremental updates.

Sometimes it gets totally corrupted and you need to nuke stuff, per your advice. This automatically leads me to the following suggestion: leave mission-critical data out of CloudFormation. Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.

Re:Two years with CloudFormation: lessons learned by SanderKnape on Sunday August 12, 2018 @10:10AM (#17744424)
You can set the DeletionPolicy attribute to "Retain" to work around this S3 issue. CloudFormation will successfully rollback without attempting to remove the S3 bucket. You can then do so manually yourself before trying to deploy it again.

Check out the docs here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...

Re:Two years with CloudFormation: lessons learned by cle on Sunday August 12, 2018 @10:15AM (#17744448)
You better do so before deploying again, because the roll forward will break since the resource already exists.

This a major pitfall when using DeletionPolicy=Retain with named resources. It breaks seamless rollbacks/rollforwards. If you rollback, in order to deploy again you need to either delete the named resources with DeletionPolicy=retain that were rolled back, or update your template to rename them all. It is such a huge pain.

Re:Two years with CloudFormation: lessons learned by SanderKnape on Sunday August 12, 2018 @10:30AM (#17744494)
True, but it beats the alternative where CloudFormation deletes objects that you didn't want deleted. The underlying issue is that the S3 objects are outside of the CloudFormation scope, thus it takes no risk and doesn't delete your objects.

A nice feature would be a "ForceDelete" deletionpolicy where it would delete the objects. You can even set this initially when creating a stack, and change it to "Retain" later when the stack is stable.

Totally agree btw that it's a huge pain initially, though once you know it it's also not that hard to work around.

Re:Two years with CloudFormation: lessons learned by cle on Monday August 13, 2018 @05:06PM (#17753640)
My preferred behavior would be for CFN to not barf when rolling forward. In other words, to be able to assume control over a resource that already exists.
Re:Two years with CloudFormation: lessons learned by scarface74 on Sunday August 12, 2018 @02:43PM (#17745768)
Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.

There are ways around it as stated below, but I agree completely. I don’t bother with CloudFormation with cross shared, building block type infrastructure like RDS and ElasticSearch. It’s just not that much of a pain to spin up a database on each account. Besides, the characteristics of the databases in different environments are going to be so different, that you are going to either have parameters or FindInMap functions anyway so for all intents and purposes, you’re not running the same template anyway.

As the article said, changing any resource that’s exported from a CF template is such a pain, it would be better just to use parameter store if you can get away with it.

Re:Two years with CloudFormation: lessons learned by auslander on Sunday August 12, 2018 @08:44PM (#17747328)
> DeletionPolicy=Retain with named resources

Try to avoid naming resources ("Name" property), so there would be no clashes. Use Ref (same stack) or ImportValue (other stack) to reference created resources. If you want, you may concatenate (Fn:Join "AWS:StackName", "-public-elb"). Cloud way is you replace things, rather than keeping it. It is convenient to know, that if I delete CF stack, everything is cleaned.

Speaking of S3, its better not to include bucket resource in CF until you know what you're doing.

> stuff like RDS databases which you absolutely never ever want to have destroyed

That begs for a separate CF stack, with template creating RDS and related things only, then export endpoint in Outputs.

Re:Two years with CloudFormation: lessons learned by tangentspace on Sunday August 12, 2018 @01:22PM (#17745331)
I agree with splitting up CF stacks this way. It reduces the the blast radius, that saved me on a number of occasions when stack updates went sideways.

The problem introduced by that approach is how to manage a large number of CF stacks. First I used a homegrown Python library to manage them, then switched to having Terraform manage CF.

At first Terraform on CF was just intended to be an expedient measure to facilitate migrating everything to Terraform, and eventually we did migrate to pure Terraform. But then we started hitting all the rough edges in Terraform. In hindsight, the hybrid approach had actually been more stable and manageable than using either tool in isolation.

Re:Two years with CloudFormation: lessons learned by deboflo on Sunday August 12, 2018 @03:32PM (#17746017)
You should take a look at AWS Code Pipeline, which natively support creating/updating cloudformation stacks.
Re:Two years with CloudFormation: lessons learned by Rapzid on Sunday August 12, 2018 @07:43PM (#17747112)
Same waiting with Terraform. If you aren't automating windows installs and/or imaging, count yourself lucky :)

Terraform also lacks a lot of the extra "smarts" that CF has; like rolling updates of any kind and some other higher level automation across different services. They take very much a blue/green approach which is beyond limiting for some services.

Ran into a few broken states using Cloud Formation via Serverless project. Luckily had been in the habbit of keeping "statefull" AWS resources such as queues, databases, and other stuff in their own stacks(or Terraform) to keep their stacks complication to a minimum and mitigate the impact of the more complicated stacks needing to be deleted..

Ultimately I started going with and promote a hybrid approach where Terraform makes sense; Terraform for base infrastructure(including most stateful resources) and CF for stuff like autoscale groups and etc.

Re:Two years with CloudFormation: lessons learned by plumeria on Sunday August 12, 2018 @08:19PM (#17747236)
Updating GSIs for a DynamoDB table has been difficult for us. We ended up recreating the table in order to change the indexes.