As nzoschke recommends, we rely heavily on ECS, RDS and other managed services. We are very careful about using exotic new Docker features until somebody else has successfully used them in production. We use DNS, load balancers and regular networking for discovering containers and communicating between them.
And it all basically just works. The worst problem we've encountered is that twice a year, we deploy a new version of ecs-agent to our staging cluster and need to revert it because of issues.
For us, the biggest challenge has been setting up a good local workflow for developing Docker apps with multiple services and multiple underlying git repositories. The docker-compose tool is great but it doesn't go far enough and doesn't provide enough structure. We've open sourced some our internal Docker dev tools here http://blog.faraday.io/announcing-cage-develop-and-deploy-co... but we think there's a lot more which could happen to make it easy to develop complicated apps.
I never liked Cloudwatch Logs and recently switched it for an ELK stack (hosted by logit.io).
If the problem is Docker complexity that needs to be solved. If the problem is difficulties of running stateless apps, or distributed storage and networking these need to be highlighted and widely understood.
Using Convox or any other platform will not magically make them disappear. You still need to troubleshoot and understand what you are running. Another layer to hide the complexity can hardly help.
I strongly agree! I'm sensitive to self-serving, but while Convox is another layer, it does make problems disappear by disallowing them.
Use Convox and try to boot or deploy a docker-compose.yml that uses networking or uses volumes incorrectly. You are blocked from doing so with a nice reason why.
> If the problem is difficulties of running stateless apps
This is not the problem. In fact this is a solved problem.
> or distributed storage and networking
These are hard problems.
Networking is largely solved with AWS VPC. Other providers do this very well too. This lends to your earlier point, adding the Docker networking layer generally doesn't help the problem. I take it for granted that any new networking stack shouldn't be trusted.
Distributed storage is barely solved anywhere.
I suppose it's much relevant for those of us who are not on AWS.
At Convox we have been running Docker in prod for 18 months successfully.
The secrets?
1. Don't DIY. Building a custom deployment system with any tech (Docker, Kubernetes, Ansible, Packer, etc) is a challenge. All the small problems add up to one big burden on you. 6 months later you look back at a lot of wasted time...
2. Don't use all of Docker. Images, containers and the logging drivers are all simple great. Volumes, networks and orchestration are complex.
3. Use services. Using VPC is far simpler than Docker networking. Using ECS is much easier than maintaining your own etcd or Swarm cluster. Using Cloudwatch Logs is cheaper and more reliable than deploying a logging contraption into your cluster. Use a DB service like RDS is far far easier than building your own reliable data layer.
Again thanks for sharing your experience as a cautionary tale.
If you are starting a new business you should not take on building a deployment system as part of the challenge.
Use a well-built and peer reviewed platform like Heroku, Elastic Beanstalk or Convox.