You're right that containers are not VMs, but that's only really relevant as pedantry of technical details.
I think that what the author was trying to say (without really understanding it) was a comparison of containers to VMs as units of software deployment.
I don't think anyone is credibly using containers as a security measure on Linux, because if they think they are, they are in for several large surprises.
Rather, we're seeing the unbundling of software - it used to be that you deployed software to a physical machine with a full OS, then you could deploy it to a virtual machine with a full OS, then you could deploy the process, its dependencies and a minimal OS into a container.
I agree that Docker doesn't have a huge and profitable future ahead of it, because it's providing commodity infrastructure. Rather I think it's interesting to think about what the next level of software deployment decomposition will be, and I'd wager that it's FaaS (ie serverless).
That isn't pedantry, it is an extremely critical point and the one most people miss when figuring out Docker. Both from a security context (docker doesn't provide vm level promises about isolation) and from a resource management side (docker is really close to no overhead).
It is not uncommon to deploy containers on VMs in the real world...
VMs are literally so hard to do correctly and in a performant fashion that parts of CPU instruction sets[0], and kernel subsystems (KVM[1]) were created to make them easier to run. Containers, in contrast are literally a few flags and a bunch of in-kernel antics.
A few people, notably Liz Rice and Jessie Frazelle have given talks on how to make containers from scratch that are very illuminating for those that are interested:
https://www.youtube.com/watch?v=HPuvDm8IC-4
https://www.youtube.com/watch?v=cYsVvV1aVss
[0]: https://en.wikipedia.org/wiki/X86_virtualization
[1]: https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine
VMs are "hard" because you start with nothing save some very low-level help from the hardware.
What I should have made clearer was that I think containers are both easier and simpler than VMs. There are both less moving parts (I haven't looked, but I assume less code), and containers are easier to get started with than VMs (set some flags on some syscalls versus make sure you buy the right CPU).
I agree, I think the next evolution of software deployment is definitely heading to sub-program sizes. I do want to point out that we've seen this before, it was called CGI. It's not exactly the same, and things will be better this time (more isolation, better tooling), but if we're doing functions-as-a-service now, we were doing scripts-as-a-service much earlier.
I think the great unbundling of the future is definitely coming and in fact it's already here -- it's just unevenly distributed.
> I agree that Docker doesn't have a huge and profitable future ahead of it, because it's providing commodity infrastructure. Rather I think it's interesting to think about what the next level of software deployment decomposition will be, and I'd wager that it's FaaS (ie serverless).
This was not what I meant to get across -- Docker may have a huge and profitable future ahead of it, but is only tangentially related to the near assured continuance of containerization. Docker as a company and project's goals are different, they do more than simply offer a way to run containers, and they have for a very long time -- my point was that the literal use of the word "docker" should die, because we should be just referring to containerization in the general sense (no matter which lib you're using). It's like the "tissue" vs "kleenex" debate, in a way.
Once you have a properly setup project going and your entire build process is mostly repeatable the benefits start becoming more obvious. Yes, you can do all the same things to a certain extent in a vm but it’s really hard to keep that streamlined and up to date. Having a script that sets up your stack in a vm on both windows and Mac then runs on Linux is a pretty big maintenance nightmare. A dockerfile works with a few commands and can be added to your repo.
It’s not without trade offs but I think if they can solve the issue of debugging in a better way then we’ll really see things solidify on this concept.
This is less true these days (and on macs that had to use the docker machine "hack" it was barely ever true, per say) -- rootless containers are on the way thanks to user namespaces. For example LXC can run fully rootless containers that act more like VMs themselves (as in they will have systemd as pid 1 inside) -- kernel support, user namespaces, and File systems in Userspace (FUSE) make this possible.
IMO One of the biggest benefits to running containers is actually E2E tests -- I don't see it done as often as I should, but it has become drastically easier to run an entire postgres instance for a single local E2E test runs -- I do this on almost every project I start now, I set up E2E tests that spin up the actual world I expect in production (so all the backing services, at the version they will run at), and interact with my application -- this is a huge step forward compared to a huge wiki that has "how to set up the local test VM" -- you can spin up and shutdown these services so fast that you can use them ephemerally (I do) without much worry.
You absolutely don't need to be a cloud person to use containers. For example I use lxc on my laptop to keep different projects separate and test new software.
Here is a general tutorial in anyone wants to test lxc:
100% agree. Most developers don't even bother testing code that requires DBs and other external services because it's a pain to keep test environments up and running so a lot of the most important and buggy code slips through the cracks.
Then you have the developers who do test but just make heavy use of mocking libs and other tricks which aren't helpful in the long run since the mocks don't actually do anything.
But setting up a docker compose with the whole stack - completely reset each time the tests are run - makes integration tests so much more valuable. And usually it's not a huge deal to set up the plumbing since the docker files are often already created for production.
So it's an innovation in packaging.
Harsh. Docker is a nice way to specify how you want your container to work. Sure release Bocker (A better Docker). But Bocker will just basically be Docker: a simple set of instructions to get a machine running.
That is all we want. A few instructions to get a system up. Devs are sick of setting up machines. Demand is there.
Also, Docker itself is very much trying to be more than what it is right now, Docker swarm, compose, and basically everything Docker as a company does to improve their product offering moves them farther away from the way people are using "docker" in conversation right now.
Basically, when people say "docker image" they should be saying "container image" (for lack of a better term, at least).
> That is all we want. A few instructions to get a system up. Devs are sick of setting up machines. Demand is there.
This is pretty vague -- "setting up machines" can mean a lot of things. I want to make it clear that it's not the container runtime's job to set up a single machine, it's to enable a process to pretend that the machine it has access to is it's own. It's a subtle difference but it's worth noting IMO.
If you want to fix setting up your machines, you should be looking at tools like Packer, Container Linux distributions (formerly CoreOS, now Atomic I believe), Nix, Guix, and Linuxkit. Docker does not set up machines, it sets up processes, but happens to bring along a filesystem (amongst other things).
A linux container is not a machine (virtualized or otherwise), it is an isolated process, I agree that devs are sick of worrying about machine configuration when setting up the required dependencies to run their processes, though, containerization is here to stay -- it was around before docker and will be here after.
docker-containers and it's related counterparts are abstractions. Useful abstractions don't necessarily equate to a new line of business.
The best attributes of containers (IMO) are packaging and distribution. What business and operators need is a repeatable, easy way to deploy applications across there infrastructure. Containers are one piece of that story.
The bigger piece, and IMO, where the business viability is, is the orchestration layer. Containers aren't very useful by themselves, you need a way to get your application online. That's where Kubernetes comes in.
You need to understand large organizations and their challenges to see what layer of the containerization stack holds the most value.
Long-term, I see 'linux containers' as we know them going away. The industry is going to move to something like [1]: lightweight, hardware-assisted VM/container hybrids. But, no matter what happens at the containerization layer, the orchestration layer is the piece that adds business value to end-users (eg, not AWS or other hosting providers).
But, starting from that kind of "clean slate" state I have to say that if it takes exasperated internet posts, like your helpful comment, to explain why containers are not like VMs and how they are not like VMs... well then maybe they are not that much not like VMs to make them such a big new thing.
That goes for many things. Like, I don't get the difference between Volley and Beach Volley. One is played on the beach. So it's volley? Played on the beach?
Feel free to make analogies between two other similar sports, such as Competition Karate and Taekwon-Do.
I mean, we can talk about things that make things similar all day, it doesn't make them the same thing. Is a hamburger a sandwich? Sometimes, the whole is greater than the parts, not everything is perfectly decomposable in life.
Because one is about putting the ball through a basket and the other is about bouncing it over a net?
>> Is a hamburger a sandwich?
Yes, actually.
But, a very special kind of sandwich.
You rarely see hypetrains for old, established things :)
It's only confusing to people who are familiar just with the popularized forms of computer science terms.
Isolation and sandboxing is virtualization. In a container, the applications seem to have an operating system and machine to themselves.
A single Unix process and its address space is also a kind of virtual machine, creating the illusion that the process has a machine all to itself. Thanks to virtual memory, other processes are not even visible; they are in a different name space of pointers. That concept breaks for multi-process applications: processes are aware of each other through manipulations of shared resources like files. Or effects like not being able to bind a networking port because some other process is tying it up. The next level of virtualization is to have namespaces for resource-related namespaces in the system beyond the address space. As far as just the filesystem space goes, we can virtualize with tools like chroot. A group of applications can have their own global /etc configuration, their own version of the C library in /lib and so on. That's the beginning of "containerization".
Yeah given how often people mistakenly call containers VMs or assume they have VM-like properties (like the ability to run a different kernel), this is tripping a lot of developers up, not just lay people.
> Isolation and sandboxing is virtualization. In a container, the applications seem to have an operating system and machine to themselves.
I don't think this is quite true, generally you can isolate and sandbox something without virtualizing anything. In this case to isolate and sandbox without a loss of functionality, we are virtualizing and swapping swap implementations out from underneath an application/process. That is not always the case though -- if you unplug a machine's ethernet cable it is isolated, but you did not virtualize anything, if you deny access to a folder on disk for a process you are isolating without virtualizing, unless you mean that any kernel interaction (like blocking an `fopen()`) is "virtualization".
> A single Unix process and its address space is also a kind of virtual machine, creating the illusion that the process has a machine all to itself. Thanks to virtual memory, other processes are not even visible; they are in a different name space of pointers. That concept breaks for multi-process applications: processes are aware of each other through manipulations of shared resources like files. Or effects like not being able to bind a networking port because some other process is tying it up. The next level of virtualization is to have namespaces for resource-related namespaces in the system beyond the address space. As far as just the filesystem space goes, we can virtualize with tools like chroot. A group of applications can have their own global /etc configuration, their own version of the C library in /lib and so on. That's the beginning of "containerization".
This is a good summary of how containerization works, and the layers of isolation provided by the kernel -- the approach here is isolation by providing a fake, but that does not mean that isolation + sandboxing = virtualization, that just means that these forms of virtualization can be used to provide isolation & sandboxing.
No, not always. Why?.
At work I have a few coworkers pushing hard to dockerize (isolate?) everything.
This makes debugging when things go wrong a lot harder.
I see isolation as one of several qualities a process could have, that sometimes is valuable enough to be worth the sacrifice.
Isolation is not some absolute quality that is without significant tradeoffs.
I've found the opposite to be true. If you have proper observability into your isolated services you can more easily pinpoint the origin of a bug.
Containerization of processes definitely increases complexity but if you can take the time to understand VMs then you can (and should, IMO) take the time to understand how containers work as well, they are lighter and simpler (for example, you don't need to build a kernel or make an initrd). I would argue that people who think VMs are simpler are actually being fooled by huge advancements in tooling over the years and the fact that it's become "easy", not that it was ever simple.
I also want to point out that containers should actually make tracking down some bugs easier, but it does so in a counter-intuitive way -- it removes whole classes of bugs from ever occurring. You'll never have two programs clobber some shared folder or resource, you'll never have programs fight over dependencies, or struggle for locally-bound ports if you're running them in containers.
Containerization definitely represents an increase in complexity, but it is well worth the effort, most of the time, granted you understand the tooling.
I don't see it as VMs vs containers.
We have a good devops process to deploy onto our instances, so we rarely have resource clashes you mention (ports/directories) because none of that is ever configured manually. All our infrastructure is derived from 'scripts', so it hasn't been a problem at all.
Aside from python, I see no advantage in containerizing any of our processes at all.
As for debugging, I always forget how infuriating it is, till in the heat of the moment I have to open up a shell into someone's badly made docker image and try to use common tools to help diagnose a problem (ps, nslookup, dig, all) all missing from the wonderful little container.
It's like being on a big navy ship, stranded in the ocean because the engines broke down, but everyone left all the tools back at the base. Yay!
It's not? I didn't mean to pit them against each other in competition, I'm saying that if VMs are worth learning about and taking the time time to understand, so are containers. It doesn't have to be zero sum.
> We have a good devops process to deploy onto our instances, so we rarely have resource clashes you mention (ports/directories) because none of that is ever configured manually. All our infrastructure is derived from 'scripts', so it hasn't been a problem at all.
It seems like it was a class of problems that you have fixed with "good devops process". I'd argue that it probably was a problem at once point, and you improved your devops process to make sure it wasn't.
> Aside from python, I see no advantage in containerizing any of our processes at all.
Well I don't know your infrastructure so I'm can't comment on that. I doubt that python is the only thing you run that could benefit from containerization (which again, means limiting access to system resources through namespaces and cgroups), but if you say so then I have no choice but to believe that it's the case.
> As for debugging, I always forget how infuriating it is, till in the heat of the moment I have to open up a shell into someone's badly made docker image and try to use common tools to help diagnose a problem (ps, nslookup, dig, all) all missing from the wonderful little container.
Sounds like you could use some more of that "good devops process" you had when you set up the deploy machinery.
Also, the fact that all of that stuff is missing from the container is actually beneficial from a security point of view -- the same inconvenience you're experiencing is the same inconvenience an intruder would experience first before breaking out of the container (assuming they had the skill to do that). This means that you have another chance to catch them downloading and/or running `ps`/`nslookup`/`bash` or whatever tooling and flag the suspicious behavior. Whether you're in a VM or not, containers are another line of defense, and that's almost certainly a good thing.
It certainly was but we fixed it and it's not a problem anymore.
> Sounds like you could use some more of that "good devops process" you had when you set up the deploy machinery.
Yeah, there are people within my group that want to 'modernize' things and put them into containers willy nilly for no real reason.
We have already solved all the difficult problems that containers are supposed to 'save' us from. Many of the proposed containers would just be a single statically linked binary with a config file.
Why?
FYI, our stuff is hosted internally, so security considerations are not such a big deal.
To hear these container advocates, you'd think that till they came around no one ever managed to use linux.
I'm fully expecting linux userland tools to go away, to be replaced by custom 'distributions' with only a kernel and a docker API soon.
They're already here!
- CoreOS Container Linux (now owned by Redhat)[0]
- RancherOS[1]
- Kubic[2] (more focused on running Kubernetes, but same idea)
There are also tools like Linuxkit[3] which focus on helping you actually build images that run the containers you want and nothing else @ startup, which is pretty cool I think.
[0]: https://coreos.com/os/docs/latest/
[1]: https://rancher.com/rancher-os/
MicroVMs start faster and provide better isolation.
Meanwhile, none of this is relevant unless you're building your own cloud platform, which is a huge waste of time for most companies.
MicroVMs, containers, VMs, zones and bare metal are places to execute code. Serverless makes all those distinctions irrelevant.
Sorry if you spent 2015 getting really into Docker. You bet on the wrong horse. It's OK, this happens in tech.
Edit: if it's unclear, I don't mean Docker itself is the wrong horse, I mean containerisation tools per se are the wrong horse - and a bad place to invest your time unless you work for a cloud provider
Docker is far from betting on the wrong horse. I can build a docker container and deploy/orchestrate it however I wish - via a docker runtime, kubernetes (which can vary in its underlying implementation), microVMs, VMs, or bare metal.
But then the containers don't need to be docker either. Theres many options available there too.
Either way, there are many options available. And the horse(s) are still in the running.
As for "serverless" vs containers/other, if you can provision accordingly, containers/other can be a better option at scale. Horses for courses, I guess.
Again, most people don't need to build their own cloud platform.
Being able to reuse Docker files is nice, you've reinvented CFengine for the eighth time. That's great.
The point is Docker is yet another standard way to build boxes and contain them. Serverless / FaaS platforms obviate that need.
* obviously an isolated kernel is better than a shared one, but the point of this post (and this comment) is that Docker vs VM vs MicroVMs vs zones etc is irrelevant for companies who are not cloud providers
It does isolate a process's view of the filesystem, it does isolate a process's view of the PID namespace, and that is valuable. Let's say it's a spectrum:
raw processes -------- processes with a certain user ---------- namespace+cgroup isolated processes ---------------- VMs
Docker is certainly an improvement for processes that you don't want running completely contained but want somewhat isolated. For isolation in every sense, you want a VM (a lightweight one if you can get it, i.e. some stripped down qemu).
> MicroVMs start faster and provide better isolation.
Agreed on isolation -- VMs are hands down better at isolating, but I'm a bit skeptical about starting faster.
> Meanwhile, none of this is relevant unless you're building your own cloud platform, which is a huge waste of time for most companies.
Again, this is exactly why it's relevant, 99% of people aren't building their own cloud platform, so they don't need the full isolation of VMs -- most of the time they're just trying to prevent program A that their devs wrote from clobbering program B that their devs also wrote when they both run on the same machine, and making it easier to deploy the dependencies that come with each.
> MicroVMs, containers, VMs, zones and bare metal are places to execute code. Serverless makes all those distinctions irrelevant.
OK I don't even really know what this means, you know serverless runs on MicroVMs right? and most of the time it's actually containers in MicroVMs? MicroVMs are just stripped down versions of regular VMs, and no one is in a zone unless they're running Solaris.
The distinction between these things is still very important, unless you mean that the future is everyone just deploying functions for their applications? But even if you mean that, cold start is basically the first stumbling block you see and it literally exists because of the distinction between how these technologies work (and how fast they can be started on demand, with how much isolation).
> Sorry if you spent 2015 getting really into Docker. You bet on the wrong horse. It's OK, this happens in tech.
Is there anyone that spent 2015 really getting into Docker and isn't better for it now? The technologies that have sprung out of this part of computing are very valuable to know and are getting more valuable, not less. You don't have to install a VM to run an isolated postgres instance on your dev machine because containerization exists -- if you're still doing this you should probably look into updating your tooling.
Also, don't forget that containerization is how some of the richest and supposedly best (due to their ability to spend money on engineers) companies in the world have been handling deployment for nearly a decade -- 2015 is late for realizing containerization is a good thing, not early.
Indeed. And since we're not building our own cloud environment, because that's a waste of resources for most companies, we will share an environment with possibly malicious code so therefore need isolation.
> unless you mean that the future is everyone just deploying functions for their applications?
Yes, that is exactly what I mean.
> You know serverless runs on MicroVMs right?
Yes, that's why I mentioned them
> and most of the time it's actually containers in MicroVMs?
I doubt this - AWSs performance documentation focuses on MicroVMs as an alternative to containers, not an addition. Which makes sense as containers do less than MicroVMs.
But hey, even if the AWS MicroVM documentation is wrong, it doesn't matter. I am not building a cloud platform. I do not care.
You're right about spin up time. Open a socket and let your apps terminate with it open for a greater chance of reuse. As a FaaS user that's your entire concern with your execution environment.
> 99% of people aren't building their own cloud platform, so they don't need the full isolation of VMs
All cloud environments must provide isolation between customers, hence VMs / MicroVMs. Customers adding docker on top of that add a huge administrative overhead that duplicates the features of their cloud provider for littl benefit,
> Is there anyone that spent 2015 really getting into Docker and isn't better for it now?
Every single person whose product is not Internet infrastructure, that wrote or configured their own unnecessary custom LXC or Docker and VM environment because 'Docker changes everything'.
If a company's product is machine learning for detecting cancer and their ops person has a custom Docker/kubernetes environment they're misusing their employer for their own technical interest
>2015 is late for realizing containerization is a good thing, not early.
It is indeed. It's just that then Docker hype was at its maximum.
OK, I'd like to note that it's also the past -- CGI was (and in some dark corners still is) a thing.
> Every single person whose product is not Internet infrastructure, that wrote or configured their own unnecessary custom LXC or Docker and VM environment because 'Docker changes everything'.
Yeah but those people now have way easier to run local environments?
> If a company's product is machine learning for detecting cancer and their ops person has a custom Docker/kubernetes environment they're misusing their employer for their own technical interest
???? If your company's product is machine learning, and developers who must work on that product need to set up their environment on their local machines, docker is easier to get started with than VMs, runs faster, consumes less resources. While it might be arguable that it's easier, there are literal money savings to be had by running a docker container instead of a full VM.
Kubernetes has much more complexity and many more tradeoffs involved so I can see that being a much heavier decision.
What you're saying is that overzealous ops people who are looking to pad their own resumes should not be allowed to run amuck, and I agree with that, but docker is not the poster boy for engineering largess -- and I'd argue it never was. Companies and research groups/smaller distributions have been using containerization very productively for a long time.
Yep. Tech does that - think about centralisation / distribution every few years, maybe it'll cycle back to people caring about their own containment tech in future. But not right now.
> Yeah but those people now have way easier to run local environments?
OK, so they wasted their time on their awful custom Docker/k8s thing that runs on top of EC2 anyway, and they have a slightly better way to spin up dev environments?
The rest of the conversation is about dev environments, bare metal, VMs and containers all have their place and I mostly agree with you (obviously containers are only useful when there's a Linux kernel on that desktop, Windows and Mac are bare metal or virtualising for the most part).
> What you're saying is that overzealous ops people who are looking to pad their own resumes should not be allowed to run amuck, and I agree with that, but docker is not the poster boy for engineering largess -- and I'd argue it never was.
You understand my point perfectly. I believe that docker is precisely the poster boy for engineering largess, but this is based on my own experiences (talking to a lot of young engineers in the startup world who love wasting investor money on ops) and it seems reasonable that you have had different experiences.
I think we have a good understanding of where we each come from and can end it here. Thanks for being civil.
However, I would add that for my own personal use, it's invaluable for development work. All that work that you do _before_ your CI or deployment.
1) When I'm working with a collection of tools that I need but are a complete mess with lots of state (think: compiler tools, LaTeX, things like that), then docker image build with its incremental way of running each command piece by piece, and saving the state after each RUN, is actually a life saver. You follow the steps of some instructions, and of course, as usual, there's one extra step not documented in the manual, so you add that to your Dockerfile. You make a mistake, no big deal, just change the command, the bad state is discarded, and you get to try again. You don't have to run the whole thing all over again. And it's instantaneous.
2) When I have to work with a client's codebase, as a consultant, you'd be surprised how many projects do not have a reproducible build, with Docker or anything else. So I end up building my own Dockerfile. The number of times I've heard "but you just have to run this setup script once" -- well, those scripts never work (why would they? nobody runs them anymore). Especially when it begins with `npm` or `pip` -- almost guaranteed to fail catastrophically, with some g++ compile error, or a segfault, or just a backtrace that means nothing. For example, I recently had to run an `npm` install command and it failed with `npm ERR! write after end`. I re-ran the container again, and again once more, and then it succeeded (https://gist.github.com/chrisdone/ea6e4ba3d8bf2d02f491b4a17f...). npm has a race condition (https://github.com/npm/npm/issues/19989; fixed in the latest version). I wouldn't have been able to confidently share with my client this situation unless I had that reproducibility.
3) It's trivial to share my Dockerfile with anyone else and they can run the same build process. I don't have to share an opaque VM that's hundreds of megs and decide where to put it and how long I want to keep it there, etc.
4) It's a small one; but speed. Spinning up or resuming a VirtualBox machine is just slow. I can run docker containers like scripts, there isn't the same overhead.
5) Popularity is a blessing; the fact that I _can_ share my Dockerfile with someone is a network effect. Like using Git, instead of e.g. darcs or bzr.
By the way, you can also do nice container management with systemd and git. There's nothing inherently technologically _new_ about Docker; it's the workflow and network effects; it lets me treat a system's state like a Git repo.
Nix has similar advantages, but I see Docker as one small layer above Nix.
> The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today.
Linux containerization (using the word "docker" for everything isn't right either) is an isolation + sandboxing mechanism, NOT a virtual machine. Even if you talk about things like LXC (orchestrated by LXD), that's basically just the addition of the user namespacing feature. A docker container is not a VM, it is a regular process, isolated with the use of cgroups and namespaces, possibly protected (like any other process) with selinux/apparmor/etc.
Containerization is almost objectively a better way of running applications -- there's only one question, do you want your process to be isolated, or not. All the other stuff (using Dockerfiles, pulling images, the ease of running languages that require their own interpreters since you package the filesystem) is on top of this basic value propostion.
An easy way to tell that someone doesn't know what they're talking about when speaking about containerization is if they call it a VM (and don't qualify/note that they're being fast and loose with terminology).
All this said -- I do think Docker will die, and it should die because Docker is no longer the only game in town for reasonably managing (see: podman,crictl) and running containers (see: containerd/cri-o, libcontainer which turned into runc) .
[EDIT] - I want to point out that I do not mean the Docker the company or Docker the project will "die" -- they have done amazing things for the community and development as a whole that will literally go down in history as a paradigm shift. What I should have written was that "docker <x>" where x is "image", "container", "registry", etc should be replaced by "container <x>".