Re:How we reduced deployment times by 95%
by maerF0x0 on Wednesday August 28, 2019 @02:49PM
(#20822762)
Actually OP gave the exact reason ECS does not work for them:
> The rate at which we can start tasks restricts the parallelism of our deploy. Despite us setting the MaximumPercent parameter to 200%, the ECS start-task API call has a hard limit of 10 tasks per call, and it is rate-limited. We need to call it 400 times to place all our containers in production.
That call limit needs to scale with the cluster size
Re:How we reduced deployment times by 95%
by otterley on Wednesday August 28, 2019 @02:51PM
(#20822785)
That sounds different from the usual way customers perform an ECS deployment, which is to replace the task definition of the ECS Service with a new one and let the control plane manage it. Only a single API call is needed, and the control plane can launch replacement tasks pretty quickly itself.
Re:How we reduced deployment times by 95%
by ahmcb on Wednesday August 28, 2019 @03:56PM
(#20823468)
We don't actually call the startTask API ourselves, but when we tell an ECS service (using cloudformation) to use a new task definition (which is basically just a new ECR image tag), ECS calls startTask and various other APIs on our behalf.
Re:How we reduced deployment times by 95%
by otterley on Wednesday August 28, 2019 @04:35PM
(#20823728)
That is true. FWIW, as an experiment I built a 50-node ECS cluster and launched 4000 tasks on it. I couldn't build a single service with 4000 tasks due to soft limits, so I built 4 services with 1000 tasks each (nginx, 256MB, 128 CPU units). No load balancer.
Overall task replacement time was about 15 minutes -- which is pretty respectable, in my view. Obviously not as fast as doing a hot code swap, though. Tradeoffs abound.
I don't know about that. ECS works fine at large scale, but it's not going to replace 4000 tasks immediately. And doing so could potentially be a huge shock to your customers if you tried. (You have to take the time to gracefully drain existing connections, etc.) Nor did the author desire to implement a blue-green type deployment which would use AWS's elasticity to its fullest potential out of unspecified cost concerns.
It's not clear to me that Kubernetes would fare significantly better here. It's not without its own particular performance bottlenecks; and the issues about safely draining connections and blue/green deployments would remain the same there. The issues aren't really related to the particular deployment orchestrator, as much as the fact that 4000 containers is pretty large number by any measure for a single unit of deployment.