We used the stock AMI. I would not be surprised if we had a configuration issue, but we spent a lot of time trying to debug it and were never able to find the root cause of the issue.
yep, we were using Amazon ECS-Optimized Amazon Linux AMI, docker and agent should've matched there.
It was strange behavior, either all agents were connected or most of them were disconnected at the same time...
How did you first discover the bug? Did your deploys just start failing in an obvious way, or did your instances disappear (from ECS's perspective)?
I'm wondering if Elastic Beanstalk is hiding this from us in some way. Or maybe just has a workaround built in?
We've been running with vanilla EB + ECS for months and haven't seen this at all.
From an outside perspective, it sounds like the primary issue you referenced here (the agent disconnecting) could have been due to a mismatch in configuration between the agent and docker, or maybe just a permissions issue. IIRC, the ECS agent tries to clean up containers every few hours so perhaps not being configured correctly caused it to get stuck?