I just spent this weekend setting this up to learn a bit of AWS for a toy project. I thought I would "just" quickly drop a Rust API server image in an ECS cluster.

By the end of the weekend, I had the architecture you describe.

- Route53 alias A record -> ALB DNS name

- LetsEncrypt cert in IAM

- ALB listener doing SSL termination using the cert -> forwarding to target group

- ALB listener doing 80->443 redirect

- Security group on ALB listener allowing only approved IP ranges in (not ready for this thing to be public yet)

- Security group on ECS service only allowing ALB to connect

- ECS cluster using Fargate

- RDS instance only allowing ECS service to connect

- CloudWatch log group for the container logs

- Subnets

- Secrets Manager for pulling Docker images from private GitHub Packages repo

Did it all in Terraform, and then added GitHub Actions to the Terraform repository to do terraform validate on PR and terraform plan && terraform apply -auto-approve on merge.

Then, yesterday, hooked up GitHub Actions on the Rust API server repository build a version tagged image and publish it to GitHub packages, create a PR in the Terraform repository to update the ECS task definition for the new image, and if it passes the PR checks, automatically merge it (which triggers the Terraform plan/apply run).

It did seem complex the first time I did it, but looking back over both the AWS and GitHub Actions configuration, I wouldn't change too much. I feel fairly confident this is secure, and I understand most of the configuration options and why they are there. Something that "simplified" it for me would just become a straightjacket as I get more proficient with AWS.

IaC 101 I guess, but I was chuffed when the Rube Goldberg machine whirred away after making a code commit to the Rust repo, and two minutes later my new code was running on ECS :)

Considering writing up a blog post about it just to firm up my own understanding as well...

What you're describing is what I sold the company on last year (save Letsencrypt, that's weird but whatever).

We only use terraform for the initial burn in (VPC, 2x public/2x private subnets, empty lb, bastion, and some subnet groups) but the rest is one to one.

Fargate isn't the cheapest platform out there but it's great for "I don't have any ops people" or "I have a fraction of several ops people not dedicated to my product." It takes a lot of patching and maintenance out of the equation.

If you want to give yourself a huge resume item, hook AWS WAF into the load balancer and play with it (you can alternatively hook it into Cloudfront if you elect to implement that in front of your LB, though then you have to make sure you protect what Cloudfront is talking to).

Easy task would be to geoip limit your application to the US, Canada, and Mexico. You can verify this by running your site through uptrends and looking at what cities get 403ed https://www.uptrends.com/tools/uptime

> Considering writing up a blog post about it just to firm up my own understanding as well...

Please do; your setup looks quite nice:)

One nit/question:

> LetsEncrypt cert in IAM

Why LetsEncrypt? AWS Certificate Manager is already free and integrated, IIRC

Would definitely be interested in that blog if you write one :)

I just want to mention that you can output the CloudFormation template that it parses out by saying `docker ecs compose convert`, which is always handy. You also could use other flags to help with joining clusters, services ,etc. Not saying it's a complete production ready too, but it is nice tool to help speed up the process.

https://docs.docker.com/engine/context/ecs-integration/

Could you comment on why you are moving away from elastic beanstalk? We manage a fairly simple EB deployment but it seems perfect for our infra needs, and I'm not able to imagine why we might need to scale out of it - if any other services are required, I'm tempted to only launch multiple EB deployments instead of going the kubernetes/fargate route.

Largely I don't think it's worth moving off EB except in specific scenarios.

I'm a Rails monolith and DB migrations are not well suited in ElasticBeanstalk when running docker containers. There's no way to run a single container that 'completes' and then returns whether it was successful or not.

Currently I have a separate environment that holds a single EC2 (lots of idle time) to run migrations. It fires and the deployment process moves on. I have to hope the migration finishes before the next set of containers gets rotated out, 2-3 minutes.

That's not a good policy to run a prod env by so I'm switching to Fargate where I can run a One Off task and poll for the result.

Other benefits include not paying for that dedicated EC2 idle time. Not worrying about EC2 management. Direct access to parameter store and secrets (EB can't do secrets from Dockerrun.aws.json yet =(

Drawbacks - I lose 'Rolling Update/Deployments based on Health' and HealthD. Another big one, I lose access to the container unless you run sshd itself (don't do it).

I am running both environments side by side to evaluate but realistically the migration requirement is going to push me to a full Fargate.

I could only run the migration step in Fargate while the rest in EB - best of both worlds? But somehow that feels dirty... we'll see what I end up doing.

We have a similar setup — containers managed by Fargate — and solved the issue of migrations by:

1) having our app containers include the DB migration logic 2) on container startup, “check and run migrations” before app startup, 3) the trick: acquire a lock in postgres as part of step 2, so that only one node at a time can run migrations.

Migrations are run inside of a begin/commit, so with the lock we have reliable guarantees that 1) exactly one container at a time can try to run the migration; 2) the migration either completes or fails.

This setup has the benefit too that, if the migration fails for whatever reason, no new app containers will start in production. That is: we can basically trigger a production deploy and if it fails, the deploy halts and the previous app version remains up and serving traffic.

There are better ways of handling this, but for us at our scale, this has been both very simple and very reliable, which makes me happy. :)

As I understand it, the concurrency-safe lock is built into Rails for a while. Step 3 shouldn’t be necessary.

https://github.com/rails/rails/pull/22122

Ah, very nice! Our setup is on python, and because of some other dev tooling we have, it’s very easy for us to have a simple decorator function that grabs the advisory lock. We can apply that decorator to any python function, including the one that triggers our migrations. If on rails, seems like using the above link would be better.

I might not know what I'm talking about (our deployments don't do automatic migrations, for eg), but if I understand correctly, your problem is to synchronize migrations with deployments, is that accurate? Couldn't you use something like GitHub Actions to run migrations and then deploy to EB?

Yeah the goal is to get the steps to be: Build, Deploy & Run Migrations, Deploy & Rotate containers.

Github Actions is CI/CD and it can be used to kick off any deploy step and manage it, but it's not one to actually perform the migration. That would be like Github Actions actually running the webserver directly.

In this example Github Actions needs to kick off a `eb deploy`, which it can totally do (I personally use CircleCI) but `eb deploy` (elastic beanstalk in general) is not designed to run a short lived script on a single container AND wait for it to return.

You can run a short lived script no problem, but you have no idea if it was successful or not because once the container becomes 'healthy' the command completes.

What I'm really waiting for is the container.lastStatus:'STOPPED' and exitCode: 0. Can't do that on EB.

EB is very limiting. If you want to deviate from it's prescribed path at all, things become a giant headache.

Then there's issues with when it goes wrong. Troubleshooting is really hard, and knowing what actually went wrong requires a ton of digging around, and god forbid the environment gets stuck in a warning state. You can only download logs when it's "OK", so you're pretty much SOL if this happens. There's also the issue where you can't be sure what's running anymore. If a rollout didn't finish, what actually got updated? There's just no good way to find out, which is absolutely terrible for a deployment system.

All in all, the drawbacks + lack of options/control make EB useless for a lot of companies. Personally, I find it best to avoid EB and just go with tried and tested methods without it. That doesn't mean rolling out K8S or using fargate either. You can if you want, but there are a whole lot of other options.

This, absolutely this. If you don't have anyone on your team with sysadmin / DevOps experience then EB is an option. However when it breaks, and it does, it's a nightmare to get working again.

Terraform is a nightmare to learn for the first time but once you've wrapped your head around it, it's a thousand times better than EB. If Terraform is too complex for your needs then use CloudFormation directly instead of EB.

For us it's the following reasons:

* It's more cost effective to place all the services on one cluster. As containers can share the instance resources it's easier to increase resource utilisation (not true w/ Fargate though).

* Support for new features on EB can be quite delayed. Eg. we need support for ALPN policies, it's a recent feature and it's not even in CloudFormation yet. With ECS we just manage the LB directly and we can do everything the API allows us to do.

* More granular control. Eg. during rolling updates we can decide a floor to how many containers are active at a given time (many others things like this).

* Integration w/ SSM and Secrets Manager.

* Better IaaS support (with EB everything happens in configuration files outside of Terraform or Cfn resources).

As an alternative to Terraform, we've had great experience using Pulumi for exactly this stack. Writing Pulumi config in Python like the rest of our code has been great.

We’re thinking about doing a similar migration. One of my concerns is that the current Beanstalk infra isn’t described in code, and when I’ve experimented a little with Beanstalk in CloudFormation it’s been horrible to write - for something a lot simpler. I obviously want us to move to using CloudFormation or something like it if we migrate, how good or bad would you say using it (or Terraform if that’s what you settled on) for Fargate is?

If you're looking for confidence in infrastructure changes then I 100% recommend fighting through the 'horrible to write' CloudFormation/Terraform.

CloudF is infra as yaml and Terra is as code. CloudF also has the CDK which can use TypeScript and be infra as code that 'transpiles' down to yaml.

All options feel terrible. It's like learning a whole new programming language that takes 10 minutes to see if your code runs or which part got stuck.

But once you've got your stack working, changes are a breeze and can be done with way way more confidence. Spinning up clone environments is barely any work.

Is it terrible? Absolutely. Is it worth it? Again - absolutely.

If I had to do it again, I would pick the CDK (more powerful), but the fight would've taken longer.

Yep I definitely agree with the importance, it’s probably the most impactful technical debt we have. Clone environments is a big driving force for it too.

I guess what I was hoping is that Beanstalk wasn’t designed with infra as code in mind, but things like Fargate were, and somehow it had become less horrible. Maybe that’s just never going to happen though and it’s the nature of the problem.

Appreciate the pointer toward CDK too, I have used it a tiny bit and it definitely seemed worth exploring further.

The code differences between ElasticBeanstalk and Fargate are not very far, but JUST far enough to be annoying.

Because you've worked with Beanstalk in the past, I would personally stick with it unless I'm looking to fight 2 issues at the same time - Learning Fargate & Learning CDK/CloudF/Terraform.

Fargate is not a panacea - Elastic Beanstalk does have very cool options/features.

I really appreciate the insight here - thanks a lot :)

Have you considered experimenting with AWS CDK? The code looked far more elegant than cloudformation

I have done a tiny bit, probably worth trying to move some smaller CFN templates over to it in earnest to try it out a bit more. I can definitely see it being nicer for things like Beanstalk/Fargate.

I highly recommend the CDK here. A setup very similar to yours is just a few lines of code which you can synth into cloudformation templates

I picked up CDK & built out the same essential setup + DB, an EC2-ECS instance for ad-hoc stuff and a bunch of things to appease Config & Security Hub in ~420 lines of quite sparse python, that builds out to ~3000 lines of cloudformation yaml.

I feel CDK has the mix between optional granularity & high-level constructs just right, while this plugin looks nice for a quick MVP standup i'd be surprised to see it in use for production workloads

I also often look at these tools and don't see clearly how they are meant to scale beyond small, isolated projects. The simple examples where you let the tool take your small project and start serving requests from AWS doesn't show me how I would use it in the real world.