I think one of the powerful things that something like containers offer is a happy path that's also the easy path path and it meshes well with the principles you need for a scalable or micro-service infrastructure. With "real" servers you can build infrastructure that operates and scales in similar ways but the happy path isn't as clear or enforced. Containers make you think about persistence and state as being distinct from your compute at/near the beginning. You're punished for not doing so fairly early, even on your local machine. Your AWS instance can go away but months down the line you're not wondering what you did wrong, you're wondering what Amazon or your ops team did wrong. So the happy path that was easier to deviate from before becomes much harder with containers.

It's also easier and faster to update a container image than servers. With something like Kubernetes you can even do it in stages. Yes, you still have to keep the underlying servers updated but they just need to run containers. No testing dependencies and prerequisites. And that decoupling of the app runtime makes updating the servers themselves easier. You can create a new machine image and replace outdated ones trivially.

> don't routine security updates mean you're constantly rebuilding and redeploying these images?

It means you should be constantly rebuilding and redeploying these images.

The fewest people I've seen use Docker actually do that.

The answer I've heard most commonly so far is "uh hmm ... right, given that I see new CVEs fixed every day, scrolling by in the `apt-get dist-upgrade` I do daily on my desktop, we should probably be doing that for our Docker as well ..."

Many then refer to per-push CI/CD, not realising that holes need to be fixed regularly and not when you push.

The fewest have automation set up that rebuilds and redeploys when the upstream docker image changes.

Plain Debian/Ubuntu servers have the benefit of `unattended-upgrades`, but with Docker you have to take on that task yourself and build automation for it.

Also have a look at https://hub.docker.com/r/library/ubuntu/tags/

At the time of writing, every single image is labelled with "This image has vulnerabilities".

You probably shouldn’t be using Ubuntu base images anyways. The closer you can get to “scratch” the better, and the fewer security related issues you’ll have. For most use cases I think Alpine is a much better base image.

That's more about trust and faith in the maintainers of the distribution (that they won't screw up).

It's about having the minimal number of system packages and libraries that your app actually needs.

Alpine is leaner so attack surface is thinner.

This. If you start adding stuff to "FROM scratch", you are creating your own obscure Linux distro. When you screw up, there are no other customers to report bugs to you, much less other maintainers to help.

You really shouldn't be adding stuff to "FROM scratch" unless you have one statically linked binary and maybe some config files or something like that. If you actually need packages, you should use something like Alpine if possible.

> At the time of writing, every single image is labelled with "This image has vulnerabilities".

The majority of those are not applicable in any way to your average container. I would highly encourage you to not attempt to make points off such bad data. See below for why I think it's bad data.

I'd also like to point out that those CVEs are even more of a pointless thing to make such a point with since almost all of them aren't fixed in upstream ubuntu... which is to say an ubuntu server with 'unattended upgrades' would be just as 'vulnerable' as these docker containers, except moreso because more of them would actually be relevant in such an environment.

Your overall point is valid, but your reference to those 'vulnerabilities' is egregiously misleading.

Let me analyse as a human all the so-called "critical vulnerabilities" listed there for Xenial:

1. glibc 2.23-0ubuntu10 - CVE-2018-6485

This is exploited by C code which intentionally calls posix_memalign or aligned_alloc (instead of malloc) with unusually large arguments. There are very few codebases out there which make use of that in the first place.

If your container is not running untrusted code which links against libc, you have little to fear. This CVE will not impact the average container running some ruby or nodejs application.

Ubuntu also offers no update yet, so it's not actionable.

2. ncurses 6.0+20160213-1ubuntu1 - CVE-2017-10684, CVE-2017-10685

Only affects you if you're piping un-sanitized user input into an ncurses application which then displays it. I doubt there are many, if any, server applications that do this.

Also not actionable.

3. shadow 4.2-3.1ubuntu5.3 - CVE-2017-12424

I'm sure there are plenty of containers out there shelling out to "newusers" with totally unvalidated input. Highly critical I'm sure.

4. cryptsetup 1.6.6-5ubuntu2.1 - CVE-2016-4484

This one only impacts the initrd of luks encrypted setups.... literally impossible to accomplish in a container. Completely garbage listing, no value, cannot possibly impact a docker container.

5. systemd 229-4ubuntu21.2 - CVE-2018-6954

This requires 'systemd-tmpfiles' to run after an attacker has manipulated the filesystem. 'systemd-tmpfiles' is not run in docker containers, and even if it did, since it starts from a fresh root filesystem each time it's a moot point since any changes the attacker makes won't persist to the next time the container "reboots" and tmpfiles runs (assuming the rare case it's run for some reason as part of the container's boot up).

Another non-applicable one.

6. util-linux 2.27.1-6ubuntu3.4 - CVE-2018-7738

This one's a bug in bash-completions for umount, which aren't installed in the container by default. This one's a false positive because the vulnerable code (the bash-completions script) is not present in the docker image.

You are are right with all you're saying here.

I was not suggesting that `docker build` executed at a given time point is less secure than `unattended-upgrades`. My point in referencing the vulnerabilities was to simply show that there is a constant stream of vulnerabilities that you need to keep patching, and that picking a new base image "every now and then" isn't enough. `unattended-upgrades` just makes it trivial to automate following this constant stream of updates, while with Docker you have to manage that yourself.

Yes, most CVEs don't affect your use case and operations, independent of via Docker or full OSs. But every now and then there is a severe CVE in that stream that affects you. You don't know when it's coming.

There are two ways to be safe: Automatic upgrades, or reading through / subscribing to the CVE stream and analysing everything that passes by (as you demonstrated here; that takes real effort and you need to be awake when it happens). Most people don't do the latter.

who has an app that they don’t deploy daily?

Some of us have contractual obligations not to deploy during the week, and require us to notify and receive permission from our customers for out of band deployments.

Many - and our deploys on weekends are almost zero.

I highly doubt that anyone would deploy everything daily. Your own code, maybe. But what about dependencies? Do you upgrade your Postgres container daily?

Full-disclosure: I’m a Consulting Architect for Red Hat focused on OpenShift.

No.

In Kubernetes and OpenShift you can control whether builds and deployments are automatic or manual, & which events triggers them [0][1]. Combined with fact that each application’s config is an independent object, this allows an admin to host hundreds of apps on a single cluster node. Usually you’re using IaC practice and storing each app’s config in a git repo or next to the app in git.

[0] https://docs.openshift.com/container-platform/3.9/dev_guide/...

[1] https://docs.openshift.com/container-platform/3.9/dev_guide/...

OpenShift has an answer to what GP was asking, but build triggers are not it.

Security vulnerabilities in images is the concern, OpenShift handles this through ImageStream('s) - parent images are tracked in the integrated registry and when one is updated all dependent images are updated.

Good example, the dotnet image in our OpenShift cluster was updated late last week - all of our .Net Core projects were automatically rebuilt with the latest image with no intervention from the developers. It doesn't handle your application INSIDE the container, or if you build the image yourself via Dockerfile so you'll still need some dependency scanning tools and/or release notifications to keep your own stuff up-to-date and secure.

To be honest ImageStream is one of the best value-add features OpenShift has over vanilla Kubernetes, I don't have to worry if developers are keeping their images up-to-date when some vulnerability in a CentOS package or application runtime gets patched.

Kubernetes itself is not a PaaS though. It won't do all of that for you, which has the advantage of being more flexible but the disadvantage that nothing happens if you don't code it.

You can automate the build process and the deploy process. Given that you can do that with any orchestration software it gives you the ability to roll back changes in your environment.

Is it more efficent than a real server? I guess thats only the case if you automate almost everything and use CI and CD.

It sure shows a diffrent viewpoint on servers focusing more on services or containers than on servers.

Comparable with Functional Programming and Object Orientated Programming. While you can archive the same functionality with both it gives you another mindset to solve problems.

I don't think people primarily choose Docker because it's more efficient(?) to deploy than a real server. The benefits of Docker, to me, are having a single artifact and reproducible builds where a developer can run the docker image locally, it then gets built once on Jenkins, and the same image is deployed to staging, production, etc. It eliminates an entire class of problems related to the operating system, installed dependencies, what language the application was written in, etc. That being said, those are also benefits when deploying, because everything is homogeneous at the deployment layer.

As far as constantly rebuilding and deploying things, we would be doing that anyway. We do dozens of deploys a day to push new code, that means building new images every time, if one of those deploys picks up a security fix that was recently merged upstream, great.

> having a single artifact and reproducible builds where a developer can run the docker image locally

This is the key thing here. As a team grows it’s easy to get various kinds of learned helplessness. Docker, for its faults, is mostly simple enough that you can expect/insist that the team use it. Which means fewer kinds of surprises at deployment time.

We enforced that by giving our devs linux workstations with no sudo rights (iso27001 requirement), but they have access to the docker daemon, and MUST install everything they need in docker images, its massively increased tools sharing between teams, and forced people to learn docker. We have some devs starting to use kubernetes locally using minikube .

Sorry for being off-topic but do you have any links or tips on how to achieve this? My understand was that adding a user to the 'docker' group gives him 'sudo'-equivalent rights.

You can set it up so that the user doesn't have to type sudo docker. But they still effectively have root access via docker.

I guess it gives some social pressure not to do superuser things?

It looks like a package manager choice. Dpkg/Apt isn't doing by default what they actually need, so they use Docker instead.

Your devs can pass --privileged to docker run, that gives them root.

Depends who's doing the choosing. Where it takes months to provision a 'real' server, Docker saves an awful lot of time and effort.

But most people choose Docker because it's well-known, well supported, and the experience outside corporate environments is a joy.

docker containers are not reproducible.

if you built the same content on machine a and built the same content on machine b the hash of the image would differ. and the timestamp probably too

Yes, but if you build the image and push it to a repository, everybody working on it can pull it and run it.

If your referring to security updates in the docker images rather than Kubernetes itself, then not really. The rebuild should be automated for you, so it's as simple as triggering a Jenkins master build.

Kubernetes handles the deployment for you, bringing down the old pods and upping the new ones without any connection loss. It makes it a lot simpler to deploy these updates.

> don't routine security updates mean you're constantly rebuilding and redeploying these images?

Yes and no; yes, because you should be rebuilding upon upstream changes, but no, because there will be fewer upstream changes. You can reduce the amount of software that is baked into a Docker image, compared to a full traditional distribution, thus reducing your attack surface and reducing the frequency with which the image needs to be rebuilt.

> And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server?

"Real" servers are always (exception: NixOS) stateful, even if they are only used to host stateless services. Uninstalled packages leave behind old service users, buggy packages depend on specific user/group IDs that conflict in production but not where they were built, conflicting ports that are already bound. Designing for immutable infrastructure makes it easier to reason about state in production and therefore makes production easier to manage.

Basically just less rolling your own solutions. It's more political than technical.

"Devops" for me is just shorthand for we want the sysadmins on our team (or on a platform our team controls) versus a separate sysadmin org we can't control. K8S is one of many things that makes that more achievable.

That, of course, has it's obvious benefits and shortfalls.

Yes, but its less of an issue mainly because there is an orchestration layer over the top of the deployment layer.

Basically there is a thing that spins up machines, connects them to a thing that places programs on them to run. The scheduler then starts your program and maps in storage, outside world connections (HTTP reverse proxy, or a bit more advance and some level 7 routing [ie /v1 goes to progam a /v2 goes to b])

when it works it means that the developer really doesn't need to think tomuch about _where_ a program runs.

Basically its a feature incomplete mainframe clone.

One point that's not been addressed in your other replies is that Docker images should be a lot smaller than a "real" server so there should be fewer issues to patch.

If you base off a very minimal image (and make use of things like multi-stage builds to remove dev. tools from the image) you can get the package numbers down a lot.

For the ultimate in low dependencies of course you can build off scratch and put a single statically linked binary into the image.

> One point that's not been addressed in your other replies is that Docker images should be a lot smaller than a "real" server so there should be fewer issues to patch.

Could you quantify this?

It's possible to strip down a "real" server much farther than even what distros' "minimal" packages specify, which is why I wonder just how much smaller "a lot smaller" really is.

I commented on a short sub-thread [1] discussing dependency bloat/radii of distros. One commentor quantified it in bytes, though perhaps a more meaningful number in this context would be number of packages.

[1] https://news.ycombinator.com/item?id=17345982

> don't routine security updates mean you're constantly rebuilding and redeploying these images?

In my experience, no. Dockerfiles tend to obfuscate deeper dependencies. Either there's stale dependencies or uncontrolled versioning. In both cases you're running mystery meat in production.

> And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server?

The fundamental DNA of configuration managers is to take a closed world - a server - as given and try to make it into the world as desired. But this assumes you can come up with a safe, terminating and not-too-long plan for doing so. Very frequently that is not the case.

What gets done in such cases? The server gets backed up, wiped and rebuilt from a clean state.

And that's more or less what you get from a container-centric substrate over a server-centric substrate. Just as I don't care about how the JVM allocates or collects my objects, I don't care about how a container orchestrator builds my processes.

The idea predates containers, but many ideas predate their economical realisation. I am most familiar with BOSH, which does this at a whole-VM-centric level (and which, like Kubernetes, was inspired by Borg). Or the 12 Factor App, which focused on this idea from a devops perspective. But I would be unsurprised to find carvings from the 4th Dynasty outlining a similar idea.

Docker is a dangerous gamble and you can get more of an automated build system, with less devops effort, from Terraform and Packer. Avoid containers and stick with real servers “baked” by Packer:

http://www.smashcompany.com/technology/docker-is-a-dangerous...

There's no reason you can build containers in an automated, routine fashion and use those to run your applications and services on. You don't have to run containers like joeblow/randomservice - start with Alpine from the Alpine maintainers (or CentOS or whatever) and write a custom dockerfile to build your stuff.

Docker aside, your arguments regarding fat binaries are kinda off mark. Python has been able to package an app and all of its dependencies into a single zip file - which is what a jar file really is - since before Go and Clojure even appeared (see PEP 273). Single-file packaging in JS is also common - very much used for browser deployment.

But "let's change languages because we don't have a reliable way to copy more than one file" sounds a bit insane to me. My solution was Debian packages, not Docker, though.

Why would I want the overhead of a hypervisor when I can just use the process isolation features built into the kernel?

Re: your essay

I couldn't quite tell but is it correct to say that you consider Docker a "dangerous gamble" because Docker Inc. may cease to exist in its current form at some point?

I've not been impressed by docker outside of the local development experience (primarily, creating sandboxes and avoiding installing stuff on the main OS, and not having the latency of starting up a VM).

This discussion is about Kubernetes though. Docker is mostly only incidental to Kubernetes. It's a much better thought through system than anything I've seen come out of the Docker world. Docker images are adequate; Dockerfile as a build mechanism for images is deeply flawed, requiring so many kludges and duct tape to accommodate shared dependencies, rebuild on upstream change, shrinking after building, etc.

Hopefully a better system for building images to run on Kubernetes will crop up.

that was a good read, thankyou

From Saturday's posts: http://catern.com/posts/docker.html An assessment of what Docker and similar actually are, and why they are unneeded redundant, insecure technology.