The main practical difference vs. running them all in one vm is explicitly documenting dependencies - both which service relies on which package sets etc. (assuming you're strict about what you put in the Dockerfiles) and which part of your system needs access to which other component or needs to share access to which directories.
You can also easily start them e.g. via systemd unit files or via a tool like fleet, and pull the ip/port of dependencies on startup of the container and pass them as arguments. This also makes deploying them as a unit easy, but still decouples it:
If you need to move one component to a different server, or suddenly realize your foobar server needs lots of resources and want to load-balance it across multiple machines, you can easily do so by forwarding a port via stunnel or load-balancing via haproxy or what have you without affecting the actual containers.
As for your health checks, you can still have a single all-encompassing health check if you want - whether it's in one "vm" or a set of separate containers does not really change that in any way, though the bigger the setup, the more additional checks you'll probably want to add. You have X containers to monitor, but you had X server processes to monitor previously. If your use case made it ok to depend on just e.g. hitting a page on a website that depends on all X services being up, then your single old monitor still does the job.
The overhead can be made really minimal and beneficial even for a single server setup, yet help you if/when you suddenly want to scale part of it. Personally, I run about a dozen docker containers on my home server so far, and I expect that to increase several times over as e.g. every web app I experiment with ends up running in its own container, with the docker setup in a git repo as documentation of exactly which project caused me to pull in which additional packages etc., and keeping the host itself as pristine as possible. Makes me lot less nervous about distribution upgrades etc.
Load Balancer w/ Builtin HTTP Health Checks <-> Application Containers [nginx+php-fpm]
Load Balancer w/ Builtin HTTP Health Checks <-> Container [nginx] <-> Container [php-fpm] <-> Additional health check you have to build out to check that php-fpm is working correctly <-> Additional Tooling to make your process work
For every single service in your architecture you are adding an additional level of redundant complexity because it "helps you scale".
I can spin up as many app containers as I want, as often as I want without a second thought or a moment's consideration. I don't care if I have 1000 of them or 1, as long as the load balancer can handle the connections.
The scale you are talking about is when you need more than just a couple load balancers...and you aren't going to need that for 90% of projects.
If you have a health check that checks that php-fpm works, it should need no or minimal adaption to handle the new scenario. The only "new" thing is that the processes have a different parent process than usual and is running in different namespaces. If you didn't care about healthchecks for php-fpm before, there's no reason to care about them now.
> For every single service in your architecture you are adding an additional level of redundant complexity because it "helps you scale".
You see it as adding complexity, I see it as reducing complexity: By wrapping each bit in a Docker container, I gain:
- Explicit documentation of the dependencies necessary for each components. I know it covers at least what is needed, because otherwise the component will fail; as opposed to documentation which easily becomes obsolete.
- Each container is limited in what it can access, ensuring that interdependencies between the containers are documented and explicit.
- Versioning of e.g. system packages can evolve separately for each component, without needing separate (heavier) full vms.
> I can spin up as many app containers as I want, as often as I want without a second thought or a moment's consideration
That's great, as long as all components of the app use sufficiently few resources, and are easy enough to run in parallel. E.g. have fun if one of the components of this container is a Postgres install and all the instances needs a consistent view of the database. Or if one of the components needs tons of memory and you end up running out of memory even if there was no need to run more of that component in parallel.
For some apps that's not an issue. For others it is, and even starting a second instance of the full stack becomes a huge waste of resources or a huge added complexity vs. starting more of a specific component.
> The scale you are talking about is when you need more than just a couple load balancers...and you aren't going to need that for 90% of projects.
The scale I'm talking about is anything from a single, small home server, where you may want to spin up more web servers for experimentation but not need more than a single database server, to large clusters. I use Docker for 100% of my small home projects because it makes life easier and saves me time.
opendais-www1 -> fails -> shoot in head, replace.
vidarh-www1 -> fails -> shoot all of the containers in the head, replace.
At this point, its the same atomic unit I originally started with. Now, I have multiple executions to perform that I have to build out. This is the very definition of wasteful added complexity. The high level process is effectively identical but you have to do more things to make it work.
> - Explicit documentation of the dependencies necessary for each components. I know it covers at least what is needed, because otherwise the component will fail; as opposed to documentation which easily becomes obsolete.
My explicit documentation also exists in the form of a Dockerfile and my configuration files. Its a self documenting process.
> - Each container is limited in what it can access, ensuring that interdependencies between the containers are documented and explicit.
How is my container not limited exactly? Because nginx and php-fpm can access the same files? o.O I don't think this is really an argument of any kind beyond the theoretical.
> Versioning of e.g. system packages can evolve separately for each component, without needing separate (heavier) full vms.
You've never seen this go wrong before in production I take it when Person X updates a component that breaks Person Y's application I take it? I have and it is not pretty. This is added complexity to manage.
> For some apps that's not an issue. For others it is, and even starting a second instance of the full stack becomes a huge waste of resources or a huge added complexity vs. starting more of a specific component.
The reason I said "To me, for my use," in the original parent comment is this is for my usage. I don't claim to speak for everyone who might use Docker. I'm sure there are people with different requirements. My requirements involve not building the sort of products you mention in the second group.
> E.g. have fun if one of the components of this container is a Postgres install and all the instances needs a consistent view of the database.
If you are storing state in an application container, it isn't a role-based container. I'm going to assume this is because I said something that was unclear.
Please explain to me how you concluded I stored state in this container:
Load Balancer w/ Builtin HTTP Health Checks <-> Application Containers [nginx+php-fpm]
> The scale I'm talking about is anything from a single, small home server, where you may want to spin up more web servers for experimentation but not need more than a single database server, to large clusters. I use Docker for 100% of my small home projects because it makes life easier and saves me time.
It doesn't save me time and adds complexity. Maybe everyone who is arguing with me is just a better developer than me. Maybe they are just people who have never worked in an environment where every added layer of complexity leads to a higher frequency of breakage because Random Team Member X forgot something.
You have a handful of lines of dependencies to document, for example using systemd unit files. You need to do the basics once, and then reuse. What you end up with are simpler containers (no separate process monitor per container etc.) at the cost of a couple of lines of extra configuration that also happens to document a dependency. I don't see that as a net increase in complexity.
> My explicit documentation also exists in the form of a Dockerfile and my configuration files. Its a self documenting process.
It documents the dependencies of the whole container as a unit. It does not without a lot of extra effort enforce capturing the interdependencies between multiple components in the same container. This is why I like to keep the containers as small as possible.
As long as you're talking about a couple of components with few interdependencies, that's not a big deal.
But once you start piling on additional components, the complexity explodes. I once worked on a project where we were considering running round-the-clock network capture for a few days to reverse engineer the interdependencies between various components because nobody were quite sure exactly what they were any more.. That's the kind of scenarios that drives me to want to encapsulate everything as tightly as possible even if I don't need it right now - complexity has an ugly tendency to creep up on us.
> How is my container not limited exactly? Because nginx and php-fpm can access the same files? o.O I don't think this is really an argument of any kind beyond the theoretical.
As long as nginx and php-fpm are the only things installed, things remain fairly simple (see below, it seems we've had a misunderstanding). Based on that specific example, I agree it's a value judgement and not a big deal either way.
> > Versioning of e.g. system packages can evolve separately for each component, without needing separate (heavier) full vms.
> You've never seen this go wrong before in production I take it when Person X updates a component that breaks Person Y's application I take it? I have and it is not pretty. This is added complexity to manage.
I've seen that many, many times. This is complexity that largely goes away when you enforce a fine grained split into containers that are built and tested as a full unit. It's exactly because I've managed systems with 100+ VM's where people have had conflicting needs for upgrades that I love being able to encapsulate each component separately, so the developers gets the freedom to make decisions about even core system packages without stepping on each others toes (as long as the developers working on a specific component can agree, anyway...)
> If you are storing state in an application container, it isn't a role-based container. I'm going to assume this is because I said something that was unclear.
I store the state in separate volumes, but the database server process still needs to run somewhere. But it does seem that we've been talking past each other. My impression was that you were advocating keeping the full stack in a container:
> Please explain to me how you concluded I stored state in this container
I saw your nginx+php-fpm as just one example since you earlier had written "In other words, its a really simple way to deploy X identical instances of an entire service/application." To me it's not "an entire service/application" unless you also include all the auxiliary processes, including database server and anything else the application depends on.
I was making a general point about putting an application stack in a single container, not specifically addressing your nginx+php-fpm example. Since I clearly misunderstood, we disagree much less than I thought. I'd still opt for separating php-fpm and nginx into separate containers too, but as I said above, that's a value judgement.
To me that is what a role based container is. Admittedly, sometimes I package an entire one-off service inside a container to make it easy to do dev work...but not in production.
> I store the state in separate volumes, but the database server process still needs to run somewhere. But it does seem that we've been talking past each other. My impression was that you were advocating keeping the full stack in a container:
Okay so we'll chock this one up to my shitty communication skills. :)
> I've seen that many, many times. This is complexity that largely goes away when you enforce a fine grained split into containers that are built and tested as a full unit. It's exactly because I've managed systems with 100+ VM's where people have had conflicting needs for upgrades that I love being able to encapsulate each component separately, so the developers gets the freedom to make decisions about even core system packages without stepping on each others toes (as long as the developers working on a specific component can agree, anyway...)
I think we'll have to chock it up to different experiences and meaning different things when we write stuff down.
I see that problem happening when you do what you are doing and enable people to make those special snowflake VMs because of conflicting requirements. The technically integrate correctly until you get an edgecase bug that was fixed in one and not the other.
As someone constantly wearing my "ops hat", I much prefer to see them treated separately for that reason.
> There are two tricky parts though: getting both underlying processes to log to the container's stdout/stderr, and getting them both to shut down cleanly in response to a SIGTERM to the supervisor.
I was simplifying it for purposes of example. The reality is I'd have a log-shipper as well shipping logs. I don't need it to dump to the container's stdout/stderr because its all in [insert log management application here].
They shutdown as cleanly enough as-is without worrying about it too much as well.
Every instance of the app has static locations for every service it talks to so it doesn't need to care what location. Every docker container has a health check for the load balancer and registers itself with the load balancer on startup.
I don't run something like Cassandra or MySQL in docker.
To me, for my use, Docker is really a lightweight role-based VM I can put on a provisioned host of some kind [another VM, dedicated server]. In other words, its a really simple way to deploy X identical instances of an entire service/application. If any component in that instance of the application is non functional the container is dead and you route requests to a different container.
The problem with the "one process per Docker instance" logic is you need a great deal more service discovery logic as a result. There are a bunch of projects/methods to simplify this but at the end of the day it adds complexity to operations. Instead of a single health check to monitor, you have X containers to monitor. You have X containers to discover, etc. Sure, this lets you get out of running Supervisor and SSH on a Docker container...but I think it adds alot of application complexity that you can avoid 90% of the time.
SOA makes sense when you have a large deployment but if you are deploying to a cluster of 5 machines...its overkill. Many [likely the majority] of projects are at the scale of a basic cluster for redundancy.