Do you think Java .jars in Maven repos or .NET .dlls in Nuget are faring any better?
Find solid dependencies, don't use too many of them, make sure you update them, use static analysis tools and dependency scanners, etc. Same drill as for anything.
That's hilarious simply because most heavy Docker users I've interacted with basically use it as a way to build houses of cards which glue-sticked piles and piles of dependencies together rapidly.
This is largely because application complexity in some domains and expectations of development pace have ballooned to insanity but there are plenty of heavy Docker users that do this even when these sort of pressures don't exist.
I think as software progresses and more development techniques become easier to leverage and automated (correctly and incorrectly), we're getting to a point advanced statistical software packages have been at for quite some time: no matter what data you throw at the software for analysis and what analysis you choose, the software has so much experience baked into it that you'll get some result that makes assumptions what you're doing is valid that may look like reasonable outputs even if they aren't. If you do need to do some preprocessing, much of that is even automated so it's incredibly easy to do something very wrong if you dont know what you're doing but not realize it. Automation has made it much easier to fail late instead of fail early.
Docker isn't the disease, its use in such manner is just a symptom.
"Allow CRUD of names" ok, you'll have it next sprint. "Allow upload of their resumé docs" no problem, two more weeks. "Parse the docx and PDF files users upload and extract the contents and fill our db schema with relevant data". Well shit. The specs to understand either of those formats are gigantic and now I'm forced to cobble together something using more dependencies if you want it done in less than two year's time.
If, for example, a file format is really complex, it has most likely one implementation that almost everyone uses. If there is a problem with it, you will hear about it. Use that.
On the other hand, if a file format is really simple or you only need a simple subset of it, rolling your own could avoid you getting 0-dayed when the most popular implementation has a vulnerability.
vs getting hit when your vulnerable implementation becomes popular. Not an obvious choice indeed.
Which is not going to happen, because there's no point in publishing something that is tailored to your specific purpose.
Also, good luck doing static analysis on binary tools.
There's an entire area of work people call "ops" that specializes on managing that dependency. But they can only do that if it's dynamic and loosely coupled.
You make your own images, if you really care. You involve your "ops" people in that.
They manage the base images that are then used by your dev team. They're responsible for updating the base images.
Your dev team provides a CI/CD pipeline which the ops team can use to check that security updates don't break functionality.
It's not rocket science.
Docker is like shipping EAR + Websphere + OS in one package.
For jars/dlls you look up dependency scanners and hook them up to your CI/CD.
For Docker images you look up image scanners and hook them up to your CI/CD.
Hot-patching apps to use newer versions of JARs because of security bugs, outside of the update stream of the actual app itself, is not really a good idea due to the risk of regressions. Docker has basically codified the failure of dynamic linking in the real world.
Unfortunately, we've gotten very caught up with the boring infra parts (the containers and the orchestration of k8s). We too often forget about the other platform we should build that uses these tools.
People don't complain about mass security holes in Heroku. That's long been container based. That's an example of building a platform that enables keeping on top of security issues.
This isn't a problem of the tools we have. It's a problem because of how we use them and what we build with them.
That's because Heroku is mostly a closed platform that the average consumer does not have insight into.
You don't know what outdated dep the software that executes your dynos runs. As the case in many areas of life, ignorance is bliss.
If you want open platforms we can talk about cloud foundry. If a base image or buildpack has a vulnerability there is a process to update it and rebuild the containers for the running application without and app dev ever needing to know about or deal with it. This is considered a feature.
The idea is to look at the architecture, concerns, and how to handle different roles with different needs at different layers of the concerns.
I don't mean to hold up Heroku or Cloud Foundry as some amazing thing. Just that you can have an architecture that solves for these problems with containers.
And why not? They're shipping benefits (immediate productivity) and externalizing costs (eventual security breaches) onto the end user.
Containers are a building block that let us handle one concern. They are a great tool for some things. But, to expect the app dev concern of "just run my code" to be solved by them well is missing the other concern.
I think Docker does some things well. I think this works as intended. I just think that we need other things that use these building blocks better to create better solutions at the right level of abstraction.
At this point in computing system complexity, secure defaults over configuration is a requirement for a sane system.
As we increasingly convert infrastructure into code, this means infrastructure configuration requires secure defaults as well.
That the Docker ecosystem doesn't seem overlying concerned (as a core, drop-everything, #1 priority) about a decades-hard security problem (detecting and resolving insecure dependencies) seems shortsighted.
To put it another way, there are two realities:
(A) No tooling support is provided to address a risk efficiently and quickly = no one addresses the risk
(B) Efficient, effective tooling is built, maintained as platform evolves, and advertised to the community = everyone does what they should
I realize the structure of Docker dissuades them from actually addressing (B), but it seems like an existential threat to adoption.
How often are people who deal with OS packages working on kernel level vulnerabilities or are dealing with microcode for CPUs? If someone is working on libc or libcurl are they supposed to follow and track all of the kernel level issues for every place the code would run?
I would argue that someone working on libcurl is working at a different layer in the stack than someone working on a kernel. libcurl can run on multiple kernels.
The same idea of layers is true when you're talking about someone working on a nodejs app and those OS layers.
> To put it another way, there are two realities: > (A) No tooling support is provided to address a risk efficiently and quickly = no one addresses the risk > (B) Efficient, effective tooling is built, maintained as platform evolves, and advertised to the community = everyone does what they should
I think Kelsey Hightower said it best when he said that Kubernetes is a platform for building platforms. I would argue that container systems in general are platforms for building platforms. I know people who spend a bunch of time building a platform on top of k8s at different companies.
Yet, in these conversation we keep coming back to Docker or container platforms as being the platform that app developers _should_ interact with. App devs speaking out, people like Kelsey speaking up, and security issues like this all say that maybe that _should_ is wrong.
Containers are a great deployment mechanism for production. Unfortunately a lot of people seem to think that they are a substitute for package management. This was obviously (and I do mean very clearly obvious in advance from first principles¹, no hindsight needed) going to become a disaster. Most people just do not care - they are either completely unaware of the problem, or they know that in a few years' time they will either have changed jobs or be promoted (in no small part thanks to their "Docker thought leadership") and it will no longer be their problem (and they would still get to claim "Docker thought leadership" success on their résumés).
¹ Also from experience - VMware came out with "virtual appliances" in the mid-2000s that promised to do a lot of what Docker would later be used for.
https://hub.docker.com/_/centos
They also have free RHEL container images.
https://www.redhat.com/en/blog/introducing-red-hat-universal...
Existing docker images are still there, future images will continue being pushed (but only for CentOS Stream post-2021), and UBI exists for minor releases of RHEL which is supposed to be identical to CentOS anyways.
And considering the fact that Docker containers don't use the kernel provided by the client OS, the difference between CentOS and CentOS Stream is even smaller than it already was. The most significant changes between minor releases of RHEL / CentOS are in the kernel.
Using distroless images or "FROM scratch" with statically compiled app reduces the risks.
You still have to watch for your app dependencies updates but that's less work than for an entire distribution.
I suppose installing the dependencies ends up being easier if you know they'll end up in, e.g. a ubuntu 18.04 image, hence that's what people do.
You don't need docker to automate your fresh install process.
The problem is not updating your images. Running old libs on bare metal has exactly the same problem.
Updating libs on bare metal has maintainers and distributions, though. And doesn't force you to roll your own security infra within containers.
If you’re using a container based on an OS, you are installing far, far less and having only one process running. That again is a huge savings: no time securing postfix and cron, etc.
Another big advantage is that you’re shifting this work from the deployment time to build time. You do it once and can test it as long as you want before deploying it to production - no more failures because Red Hat’s mirror is unavailable or malfunctioning during your prod update window.
Even if you do a FROM Ubuntu or FROM Debian that's still a huge advantage.
nothing in docker prevents updates, just people in general are poor about doing maintenance, this is a story that is old as computers, and why MS forced updates on everyone because left to their own most people choose to "postpone" them indefinitely for fear of breaking things
It sure would be helpful if the updates didn't actually break things. It's usually not the security fixes that do that—stuff rarely breaks on LTSC, or in Debian point releases—it's all the other changes that get rolled in.
There's just too much churn across the industry. Even if the changes are good, too little thought is given to the cost of change itself.
As much as I love Docker for development, going all-in on containers for prod was a huge mistake. Which is becoming clearer now that RedHat pulls free CentOS images, after taking the so-called Linux community further and further away from POSIX and other portability means with systemd, namespaces, oci, and whatnot.