News for hackers. Stuff that matters.
Re:AWS CloudFormation now supports blue/green deployments fo... by ciguy on Tuesday May 19, 2020 @11:59PM (#23243159)
The ECS default deployment model is quite frankly a disaster. I've written many Python/Go scripts over the years to wrangle it into a sensible form for CI/CD.

Good to see that they're working on it, but I don't know why they don't fix the underlying paradigm instead of making it Cloudformation exclusive.

Re:AWS CloudFormation now supports blue/green deployments fo... by vazamb on Wednesday May 20, 2020 @12:15AM (#23243239)
I would love to know what the problem is. We do dozen of deployments every week with a ALB + ECS + Fargate setup. We upload a new container image, create a new task and launch as many tasks as desired (so if we want 2 containers running we launch 2, for a total of 4). ALB calls the /health endpoints on the new containers and if they pass the healthchecks it drains connections to the old containers and stops the tasks. This has worked seamlessly for a long time now without any downtime during deployments.

EDIT: I should mentioned that we are using AWS CDK for all of this. All it does is register a new task as the default task for a service and ECS/ALB does the rest.

Re:AWS CloudFormation now supports blue/green deployments fo... by Nk26 on Wednesday May 20, 2020 @12:28AM (#23243303)
I would like to know the same, we have moved almost everything to fargate and ecs and have had zero issues.
Re:AWS CloudFormation now supports blue/green deployments fo... by WatchDog on Wednesday May 20, 2020 @06:21AM (#23245214)
I find ECS particularly, ECS on EC2, to be really painful for small deployments.

I just want a cluster that scales in and out to the amount of memory my tasks need, I'm not very concerned about CPU. Until capacity providers it was more or less impossible to do so without having a bunch of excess capacity provisioned. Even with capacity providers, I can't seem to get a cluster to scale in to 0 instances when no tasks are needed.

I have one ECS service that requires an EBS volume mount, which means that when the task definition is updated, I need the service to stop, so that the new task can mount the same volume. This deployment model is essentially impossible without implementing a custom deployment strategy.

Fargate makes things a bit easier, and now that you can use EFS volumes with it it might make more sense. But overall, everything in ECS just seems poorly designed, clunky and hacky, like many AWS products.

Re:AWS CloudFormation now supports blue/green deployments fo... by friend-monoid on Wednesday May 20, 2020 @08:04AM (#23245841)
You can’t do EFS on Fargate in CloudFormation yet though.
Re:AWS CloudFormation now supports blue/green deployments fo... by scarface74 on Wednesday May 20, 2020 @08:33AM (#23246118)
True. But you can do it with a custom resource.

https://gist.github.com/guillaumesmo/4782e26500a3ac768888daa...

Re:AWS CloudFormation now supports blue/green deployments fo... by WatchDog on Wednesday May 20, 2020 @07:13PM (#23253108)
You also cant use capacity providers with cloudformation yet. Likely due to the fact, that you also cant delete capacity providers, nor change a cluster's default capacity provider.
Re:AWS CloudFormation now supports blue/green deployments fo... by ciguy on Wednesday May 20, 2020 @02:58AM (#23244061)
The default model works for most use cases, but it's not really flexible for non-standard cases and it fails completely at larger scale. If you have a few dozen containers taking 100,000 requests per second and you want to slow warm them with a segment of traffic for example, it's much easier to do on Kube than on ECS. ECS is also possible but it's just not as transparent or easy to work with.
Re:AWS CloudFormation now supports blue/green deployments fo... by x3n0ph3n3 on Wednesday May 20, 2020 @01:06AM (#23243454)
Same here -- I perform the exact same kind of deployment you mentioned using CloudFormation. My only grief is poor rollback detection / control.
Re:AWS CloudFormation now supports blue/green deployments fo... by scarface74 on Wednesday May 20, 2020 @08:29AM (#23246086)
That’s not a Blue Green deployment.....
Re:AWS CloudFormation now supports blue/green deployments fo... by ganstyles on Wednesday May 20, 2020 @11:21AM (#23247945)
I don't think they are saying it was, unless I'm misreading. They're talking about standard ECS deploys. Yo add my anecdata, I do lots of ECS deploys via terraform into production and it works pretty seamlessly.
Re:AWS CloudFormation now supports blue/green deployments fo... by scarface74 on Wednesday May 20, 2020 @11:59AM (#23248433)
I don’t have a problem with deploys with CF. But it only let’s you configure a minimum healthy percentage. Which is good enough if you only need to validate that an instance is in an acceptable state via a health check.
Re:AWS CloudFormation now supports blue/green deployments fo... by sandGorgon on Wednesday May 20, 2020 @12:22AM (#23243272)
Exactly this - I think internally there is a tussle on what is the strategic way forward.

There are so many - Elastic Beanstalk, ECS, EKS, ECS-on-Fargate, EKS-on-Fargate..and of course the huge marketing push for Serverless.

They could have the sensible way out and built EKS as the foundation of everything - makes total sense given the massive ecosystem around kubernetes. ECS and Fargate should be killed off.

https://cdk8s.io/ replaces Elastic Beanstalk ....but still basically runs on top of EKS.

The pairing of CDK8S and EKS are fundamentally enough for all usecases that AWS basically sells.

Re:AWS CloudFormation now supports blue/green deployments fo... by Androider on Wednesday May 20, 2020 @01:21AM (#23243530)
CDK8S doesn't replace Elastic Beanstalk, because the folks using EB are going to continue to use EB. If it ain't broke...

ECS predates EKS, and is better integrated than EKS with their other services. If you want a painless container experience on AWS, ECS for Fargate is what'll you'll want to use today. So both existing folks who use ECS today as well as new customers are going to keep using ECS.

Folks who like K8, perhaps from prior experience, are going to use EKS despite it having some rough corners.

See, Amazon is perfectly happy to support everything under the sun as long as there are paying users. See their database offering for very much the same strategy, and it would be silly to say that they should drop DB x since they now offer DB y.

Re:AWS CloudFormation now supports blue/green deployments fo... by sandGorgon on Wednesday May 20, 2020 @03:00AM (#23244068)
The different proponents who use ECS, beanstalk,etc are really talking about the UX. Because ultimately the orchestration stack is not visible to you.

And my whole point is that AWS is on the path to fix kubernetes UX and give you exactly the mental model you want...but run it on k8s.

Think of it as ECS, Beanstalk, Fargate on top of EKS.

You won't be asked to adopt the complexity of k8s

Re:AWS CloudFormation now supports blue/green deployments fo... by tybit on Wednesday May 20, 2020 @02:52AM (#23244020)
AWS doesn’t turn off barely used services, ECS has lots of happy customers that don’t want to deal with Kubernetes.

Same goes for Fargate, except that it’s now offered by EKS to, what possible reason would they have too turn it off?

Re:AWS CloudFormation now supports blue/green deployments fo... by thoraway1010 on Wednesday May 20, 2020 @01:22PM (#23249393)
Happy ECS on EC2 and ECS on fargate customer. I'd be really surprised if they turn these off - they just work in a very simple way to get stuff running. I have some "scheduled tasks" in containers, they run on fargate - works great and so easy to update.
Re:AWS CloudFormation now supports blue/green deployments fo... by sandGorgon on Wednesday May 20, 2020 @02:57AM (#23244048)
CDK8S will actually abstract away the complexity of kubernetes. I don't think you will find the complexity any worse than ECS
Re:AWS CloudFormation now supports blue/green deployments fo... by harpratap on Wednesday May 20, 2020 @01:22AM (#23243532)
I think it is a good strategy for AWS to invest in all kind of different solutions for Container technology and see which one wins. Betting on just one horse might end up being too high of a risk. Google has GKE, Cloud Run, Anthos and directly run containers on Compute Engine
Re:AWS CloudFormation now supports blue/green deployments fo... by solidasparagus on Wednesday May 20, 2020 @01:17AM (#23243514)
k8s is not for everyone. It has a high administration and complexity burden. It would be a mistake to make k8s a requirement for core AWS services. Fargate is a low-level compute platform - it is parallel to ec2 and lambda. It does not compete with kubernetes (as can be seen by the eks-on-fargate offering).

There is probably some tension between ECS and k8s - AWS built a container orchestration platform based on what they think the world (and Amazon) needs and then k8s became madly popular. And it's not clear that AWS was wrong because k8s is essentially too complex for many use cases. It makes total sense that they would support both fully.

Re:AWS CloudFormation now supports blue/green deployments fo... by harpratap on Wednesday May 20, 2020 @01:42AM (#23243626)
Vanilla K8s is not meant to be directly used by everyone. Watch https://www.youtube.com/watch?v=ZqQTEdHVaCw for insight on how K8s team expect the project to move forward. Frameworks like OpenDeis and Knative are meant to be used by developers.

IMO K8s will become akin to Linux Kernel. Almost no one uses bare mainline Linux kernels, you choose a distro based on your needs. Companies like RedHat and Canonical will pop up and provide their own packaged "distros" of K8s and you will choose which philosophy best suits your needs.

Re:AWS CloudFormation now supports blue/green deployments fo... by bdcravens on Wednesday May 20, 2020 @12:37AM (#23243339)
We haven't had any problems, but our workload is mostly async background work. Our CI pushes master when it passes, running a simple script to push to ECR with a git SHA tag, update the task to use the latest image, and then update the service to the latest task definition. It takes 2 lines of bash to update the task definition, and 2 lines to run the AWS commands.
Re:AWS CloudFormation now supports blue/green deployments fo... by ciguy on Wednesday May 20, 2020 @03:00AM (#23244065)
Yeah for the standard vanilla case it's ok. As soon as you hit a certain scale or need to do anything non-standard it becomes much harder to work with.
Re:AWS CloudFormation now supports blue/green deployments fo... by snypox on Wednesday May 20, 2020 @02:11AM (#23243789)
Really? At the last company we had like a 200 lines PowerShell script for all of that.
Re:AWS CloudFormation now supports blue/green deployments fo... by scarface74 on Wednesday May 20, 2020 @08:37AM (#23246152)
You can do all of it by pushing the container to ECR, tagging it with your build number and running CloudFormation. You pass the build number in as a parameter and you specify your image as

  image: !subst “image:${Tag}”
Re:AWS CloudFormation now supports blue/green deployments fo... by justicezyx on Wednesday May 20, 2020 @12:04AM (#23243182)
Could you elaborate a bit on the problem of "ECS default deployment model"?
Re:AWS CloudFormation now supports blue/green deployments fo... by ciguy on Wednesday May 20, 2020 @03:02AM (#23244085)
It's just very rigid and not easy to extend. As soon as you hit a certain scale or need to do something slightly non-standard it's a mess.
Re:AWS CloudFormation now supports blue/green deployments fo... by tbrock on Wednesday May 20, 2020 @08:16AM (#23245951)
Really? We’ve been using it for 3 years and it works fine. Make sure you pay attention to min/max task settings.

There are basically three ways to do it:

Min less than max, max is 100.

Max greater than min, min is 100.

Some combination of the two.

For the first you take down running tasks first and then backfill with new (green) tasks.

For the second you add green tasks and once stable take down blue.

Make sure that if you only have two tasks that min/max move in 50% increments: I.e you can’t scale up/down to 125%/75% but can to 150%/50%.