ECS has a docker size limit of 4GB before it will crash silently with no user facing error message. It is also is pretty buggy with the agent sometimes not starting and paying for nodes that aren't used.

Also containers are better for production security than just having apps sitting side by side on the same disk.

ECS is simple to get started, but does not have a vast array of features that are necessary to run a service reliably at scale. It is possible to add many of these to ECS, but it’s left as an exercise to the reader to write up various API’s, cloudwatch, Lambda, etc.

I can provide more specific details if there’s questions.

Thank you. Yes, could you name the features ECS is missing for running at scale reliably, please?

I've recently moved from ECS to Kubernetes, and much prefer kubernetes

In general ECS is fine when you have 1 or 2 services, but once you have 10s or 100s it gets pretty unusable.

Logging: While ECS ships everything to cloudwatch, actually accessing them is a nightmare and usually more effort than it's worth. It's very easy on k8s to get logs shipped from every container to elasticsearch and browse with kibana.

Autoscaling: Possibly less of an issue now that fargate's a thing - but fargate's expensive. Kubernetes makes it easy to set up both node level autoscaling, and deployment level autoscaling

Ingress: With ECS you have to set up loadbalancers for every service individually - in k8s once the controller's properly set up every service defines its own ingress, and the only manual change you have to do is create a DNS entry (only if you can't use a wildcard)

Metrics: Prometheus is very ingrained in kubernetes, and you get a wealth of information from every service almost "for free" once you've installed it.

And more: service meshes, secret control (through e.g. vault), declarative definitions, provider agnostic

Kubernetes has a huge startup cost and learning curve, but is incredibly powerful once you're up and running

Autoscaling: No need for Fargate. Create 'AWS::ApplicationAutoScaling::ScalableTarget' and 'AWS::ApplicationAutoScaling::ScalingPolicy' for your 'AWS::ECS::Service' resource.

Logging: Subscribe the ElasticSearch Lambda to CloudWatch Logs group to have all logs in ElasticSearch domain. Kibana is included in 'AWS::Elasticsearch::Domain' resource.

Ingress: Yes, one ALB per one autoscaled ECS Service, I don't see it as a problem.

Metrics, And more: not sure I see the benefits. CloudWatch has miriad of metrics, you can get lost there :)

It might look hard, but here is great starting point : https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui....

Yes, if you want to use AWS, use Cloudformation templates, there is no way around it :)

I couldn't figure out how to run master-slave configuration on ECS.

We've had good luck with ECS (though occasionally I'll find it doesn't shut old tasks down). Honestly though I feel like ECS will go the way of services like SimpleDB, Cloud Search, etc. (Never going away, but a secondary offering)

Law of natural selection. Good thing is AWS would never depreciate an actively used service.

Orchestration proved to be difficult in ECS for me.

If you're just launching a standard app, pretty easy. However, anything complex requires writing a ton of AWS SDK code. (For example, we launch a spot instance with a given task definition, assign a given customer's work to it, then shut it down when the work is finished, and then on to the next customer)

Too complex, KISS principle rulez :) Creating ECS Cluster on spot instances seems like a way to go. Haven't tried myself yet, in to-do's :)

We had a simple setup before (and have used spot instances everywhere it made sense). Without getting into the specifics of the use case, the ability to target a specific instance for specific backend processing tasks became necessary. Rest assured, it's not "too complex", only "as complex as necessary". Be careful not to respond with a rote "you're doing it wrong"-ism when discussing technical merits.

Can you elaborate? Can you give an example of a specific problem you ran into with ECS, and how you were able to easily solve it with K8S?

I couldn't get my nginx container to communicate with my app container in ECS however it all worked locally. And theres very little docs or troubleshooting resources.

That's a rather bold claim, care to elaborate?

Yes, happy to. Which claim you would like to focus on?

Not the person that originally asked you, but I'm interested in hearing why ditching docker is needed for high volume production. First, what do you consider high volume? After that, what's wrong with docker within that definition?

Maxing out your CPU and networking. If you're there, you'll be running many hosts under load balancers. Be it cloud instances or physical machines. Here you have no need for containers. Web servers and backend apps are capable of multi threading.

You can't just "enable multi threading" in a varietal ecology of software applications and contexts.

Also, this is not an argument _against_ containerization. You make no argument or show any evidence of performance benefits bare v. docker.

We max out our compute about 3% of the time, and run at about 10% load the rest of the time. It's a similar story for memory consumption. And we run this multitenanted, where the max doesn't overlap for different tenants. And we can't move our customer data out of our data centre, so we can't use cloud elastic compute.

It's in this kind of a world that elastic compute provided by a cluster scheduler with efficient packing looks really appealing. And our jobs, while meaty in time and space, are stateless once the initial data is poured in, up until the results pop out - they are big functions, basically. Good match for a stateless container.

So k8s looks quite attractive.

That's ... not even wrong.

Containers not being good for production.

Excellent point :) So. First of all, you'd not have to worry about following, if you don't use containers.

If your app, running in Docker container, is compromised, like PHP webshell, it might try to escape the container. What capabilities you granted for your containers? CAP_SYS_ADMIN, CAP_NET_ADMIN or you have no idea? This is just one example of escaping https://www.twistlock.com/2017/12/27/escaping-docker-contain... and lets talk about namespaces, like user namespace. Is root inside container is a root user on host system?

Does it feels like production ready system?

Most operators lock down to the runtime pretty tightly, leaving very few capabilities enabled.

If that's not enough, it's easy enough to ensure that a Pod is scheduled to run by itself in an otherwise unoccupied VM.

If that's not enough, the IaaS providers can be paid extra to ensure yours is the only VM on the physical machine.

You can have the same expensive guarantees, if you need them, but with a uniform control plane for all workloads. That's pretty attractive.

Yes, it does.

1. Every system has vulnerabilities. You can defend against them.

2. Any improperly configured system can be abused. In particular, the exploit you linked can be completely stopped with a litany of ways. https://news.ycombinator.com/item?id=16030107

Your argument going from "containers are unfit for production, you'll mature out of them one day" to "here's a small, preventable vulnerability" seems more like a security non-sequiter than an actual argument against containerization.

Further, claiming containers are not production ready is empirically and literally negated by them used, in production, by the largest tech companies that have ever existed.

If the same compromised app is running natively then the entire system is now compromised. Capabilities and namespaces can be used with or without containers.

1. You may have different apps, owned by different owners, running as containers on same underlying host machine.

2. Not running docker means you can lock your httpd by chroot, FreeBSD jail or OpenBSD pledge.