"If dev and prod run differently" — Why is there an "if" here at all? Isn't it obvious to verify with two eyes in the real world? Is there any company in the world that runs production from somebody's laptop at home? Or distributes server racks for people to take home? It looks like an extremely disillusioned question out of touch with reality.
1. For development, engineers want to run a local throwaway mysql/redis. In production, engineer want to use a proper managed mysql/redis. The compose file, and envvars/links will be different. This is extremely normal, it makes sense.
2. Development laptops will always be underpowered than servers. The cpu/memory requirements will be different. Again makes sense.
3. For development, engineers will build and run the docker image locally. In production, these are separate, you would build and publish the image separately in CI, and run only published images in prod. Again, makes sense.
These are all perfectly logical things that make practical sense.
Trying to share/ reuse/ resemble is fine. But saying "dev===prod" is just a disillusion out of touch with reality.
Edit: For context, yes I do docker for a living for last 10 years, since v0.4 (lxc) days. I've been to and presented in meetups. If someone says they're trying to get dev as close functionally to prod, by doing X, Y, Z and sharing practical tips — you know you can listen to them. If someone says "dev===prod" and when you ask how they talk abstract principles, you quietly get out of there.
We haven't achieved it (yet?) but that's the direction I was aspiring to!
Bad idea. You develop against a completely different local system, then you push to prod, and "oh no it doesn't work". Yeah - because you're developing against a completely different system! It behaves differently. It causes different bugs. You take a ton of time setting up both different systems differently, duplicating your effort, duplicating your bugs, having to fix things twice for no reason, and causing production issues when expectations from local dev don't match prod reality.
Every experienced systems engineer has been repeating this for decades. Listen to them. They all say exactly. the. same. thing. "Make all your systems as identical as possible." That doesn't mean make all your systems different whenever it is more convenient for you. It means that if it's possible for you to be developing using the same tech you use in prod, you should be doing that, unless it is impossible. If you think it's not possible or practical, you simply haven't thought hard enough about it.
The whole selling point of the cloud is that you can spin up environments and spin them down in minutes and pay pennies for it. Not a shared environment where everyone's work is constantly conflicting, but dedicated, ephemeral environments that actually match production. Literally the only example I know of where this doesn't work is with physical devices where you have a lab of 10 experimental pieces of test gear and you have to reserve them to test code on them. That can make remote development difficult. Still not impossible though.
> The cpu/memory requirements will be different.
Unless you're developing using the same servers as prod. There is no reason that you have to use your laptop. If you think "it's more convenient!", that's only because you have done zero effort to make Prod-like ephemeral environments more convenient.
> For development, engineers will build and run the docker image locally. In production, these are separate, you would build and publish the image separately in CI, and run only published images in prod. Again, makes sense.
That is the antithesis of what Docker was created for. The point of containers was to say "it works on my machine", and use that exact same image in production. Not a different image in CI that nobody developed their code against. It's just going to result in bugs that the developer didn't see locally, so they will need to spend more time to get it working. Your laptop can do the same build CI can on the same branch and push the same image that CI can.
What I'm saying is nothing new. Look into cloud-based development environments and the dozen different solutions made solely for rendering your local code in a remote K8s pod as soon as you write to a local file. Imagine a world where you don't waste your time testing something twice and fixing bugs twice.
>If you don't use the same Docker Compose file for Production, don't use it for Development. Your two different systems will diverge in behavior, leading you to troubleshoot two separate sets of problems, and testing being unreliable, defeating the whole "it just runs everywhere" premise.
Anyone ever developing a react project on docker compose would never run a build step for development.
In fact for most setups, I probably wouldn't even run react on docker-compose for development, and just use the dev server straight up on my machine.
Furthermore, working with any python http servers is substantially easier when working with a dev server in development, rather than running your server through gunicorn or other wsgi servers - not to mention the ability to hot reload for development.
I'm sure that there are many other cases where it's necessary and convenient to separate production and development docker compose files.
I don't think it's fair to say that there is no point in having separate docker-compose files as the lion's share of dependencies that need to be consistent is inside each container, not on the docker-compose configuration level.
I then find out that no-one bothered to run the compose file that matches what gets put into prod on their local machines.
My point is: your view might become more black and white on this matter after the N-th "but it works fine on MY machine" comment.
EDIT: Where N = your personal tolerance level of bullshit.
There's always one in the crowd that refuses to give up local development.
But even so, how you can go from that to:
> [...] there's really no point to using Docker Compose other than developing without internet access
boggles my mind.
EDIT: apologies, mixup on my end and quote is from someone else.
EDIT: I usually try to build, test, inspect, push and deploy from the same compose file. You can deploy to ECS direct from compose using an AWS context: https://aws.amazon.com/blogs/containers/deploy-applications-...
Straight up 100% hard agree.
> It goes without saying that Docker Compose only works on a single host, so that won't work for Prod if you need more than one host, but single-host-everything seems to be HN's current fetish)
technically this is only correct for compose file versions up to and including 3.8.
Part of the push with the new mainline compose V2 plugin has included the updated compose specification [0] which seems to have unify quite a lot of differences between the "compose" file for compose deployments and the "stack" file for swarm deployments.
FYI You can switch over to the new version with the compose V2 plugin by remvoing any `version` key from your compose file.
Although the `deploy: restart_policy:` values are currently not unified, much to my sadness.
I've heard of and worked with multiple dev teams that develop with compose and deploy a totally different way (k8s, AWS auto scaling groups, heroku, etc), to great success.
Devs know about compose and understand it. As long as the containers behave as planned, the DevOps people can figure out how to deploy them on whatever production environment.
Managing all these different configurations and the matrix of interactions between them is why we're trying to integrate the SDLC into one configuration at Coherence (withcoherence.com). [disclosure... I'm a cofounder]
I am now using this setup[0] for development, but it doesn't really match production 1:1.
Docker Compose is fine for running smoke tests/unit tests, but if Dev and Prod run differently, there's really no point to using Docker Compose other than developing without internet access.
(It goes without saying that Docker Compose only works on a single host, so that won't work for Prod if you need more than one host, but single-host-everything seems to be HN's current fetish)