News for hackers. Stuff that matters.
Re:Docker Image Insecurity by shykes on Tuesday December 23, 2014 @12:53PM (#8789181)
I wish the author had not omitted this crucial paragraph in the announcement he quotes:

    Note that this feature is still work in progress:
    for now, if an official image is corrupted or tampered with,
    Docker will issue a warning but will not prevent it from
    running. And non-official images are not verified either.
    This will change in future versions as we harden the code
    and iron out the inevitable usability quirks. Until then,
    please don’t rely on this feature for serious security, just yet.
So, we've made it pretty clear from the start that we're working on ways to make image distribution more secure, but are not claiming that it's more secure yet.
Re:Docker Image Insecurity by Titanous on Tuesday December 23, 2014 @01:05PM (#8789255)
Author here.

> I wish the author had not omitted this crucial paragraph in the announcement he quotes:

I did not quote this announcement, only linked to it.

If this is intended to be prototype code, it should not be enabled by default, and certainly shouldn't be printing inaccurate messages to users in a 1.0 product.

Also, the referenced warning goes to the daemon log, and doesn't appear to ever be triggered.

Re:Docker Image Insecurity by jsmthrowaway on Tuesday December 23, 2014 @01:22PM (#8789356)
I wish you'd take things like this seriously when they become public and not spend your time telling the world how everyone is wrong about Docker. The author spent a lot of time disassembling this issue and he doesn't even work for you; rather that interpret that as trying to make Docker better, which it is, you're right back on HN playing the "nobody understands what we say and everybody bullies us" attitude that Rocket really brought out of you. I wish you wouldn't do that.

We can't all get what we wish for, apparently. The difference between your wish and my own is that the less I get mine, the more you alienate the people in this community. It's your prerogative not to care about that, but how you act on HN has really made clear to me that Docker is riding on simply being first, just as MySpace did.

Re:Docker Image Insecurity by shykes on Tuesday December 23, 2014 @02:27PM (#8789740)
We do take security reports very seriously, when they were brought to the project repo: https://github.com/docker/docker/issues/9719

If you are looking to gauge our reaction to legitimate criticism and bug reports, you should follow the github repo and mailing lists, not hacker news.

Re:Docker Image Insecurity by jsmthrowaway on Tuesday December 23, 2014 @02:56PM (#8789876)
I'm not talking about Docker. I'm talking about you. Your first inclination is how people are wrong and only you are right, in every case. I don't understand your need to make everyone who criticizes Docker feel wrong, or like they missed something. Your thread here about Rocket was extremely alarming and directly motivated me to plan for a future without Docker.

Best way forward here would have been "hey, thanks for looking into this, we know it's an issue." All of which you said later after making sure the author knew he missed something, which no normal user is going to look for anyway.

The real bummer is you remind me of the CEO of Linode, whose company will fail for similar reasons. He is equally infallible and ignores criticism.

Re:Docker Image Insecurity by shykes on Tuesday December 23, 2014 @03:56PM (#8790171)
That's an understandable criticism, even though it makes me want to die inside. Let me try and explain my point of view, and why this whole situation is so frustrating to me.

First, my impression is that I do spend most of my days listening to criticism by people smarter than me, revisiting my assumptions, and trying very hard to make people feel appreciated for their effort in the process. Most of that effort is spent on IRC, the Docker repository, and the Docker mailing lists (in decreasing order of time invested). Obviously, I am biased. But I'm willing to bet that if you spent a week or 2 on the IRC channel interacting with me and the other maintainers, you would develop an entirely different picture of how we work and how we treat criticism.

Second: aggressive blog posts by competitors and their coverage in Hacker News. This is something new to me, and I acknowledge that I have been handling it very poorly. I thought I had learned my lesson last time, but as your message points out, it appears I have not. Basically, I handle very badly the combination of 1) feigned indignation and ignorance by people with clear conflicts of interest; 2) a small number of vocal haters waiting for an opportunity to pile on, and 3) lots of intelligent but uninformed bystanders. When someone makes a valid criticism, with the ultimate goal of improving the project, as harsh as it is, I will be the first to embrace it. But here we are talking about something different: a blog post written by commercial competitors, with the intent of making the project look as bad as possible - using valid known criticism when available, but not shying away from recycling old criticism which has since been addressed, or sometimes making up facts altogether - resulting in a more compelling narrative that will "stick" better. These are posts crafted by venture-backed companies competing with Docker, inc. in the same general space (developer and IT tools), with all sorts of conflicts of interests. I now these people personally, they are all part of the Silicon Valley scene, and what they do is very intentional. It has nothing to do with honest criticism or attempts to improve the project - those are taking place all day long on the repo, and never generate any drama. It's just business as usual, but as an engineer it drives me crazy. What drives me most crazy is how easily people fall for it. But, I acknowledge that it's not an excuse for the sort of knee-jerk reactions I've engaged in. As frustrating as it is, I think the proper reaction would be to 1) stay under the radar, wait for the drama to die down; 2) take a solid look at the criticism, extract the real one from the political bs, and focus on fixing that; 3) take a solid look at how well we communicate what we do. For example, we have made good progress on security compared to 6 months ago, but clearly we have done a poor job at explaining that and showing solid proof points. 4) as outrageously hypocritical and unfair as these posts appear to me, who know the full back story and understand the commercial motivations, remember that the rest of the world is not aware of the commercial back story. And frankly, they don't care. What they'll remember is our response to the criticism.

So - I did my best to express my point of view on all this. My personal lesson is that success is not that fun, and sometimes I regret the days when Docker was not successful. But at the same time there is a sense of responsibility: a great many people use this tool every day, and we are responsible for improving those tools every day. Once these vc-backed competitors (whether it's coreos or flynn) are done writing their blog posts, they aren't going to fix the project. That's our responsibility, and nothing else should matter.

So, for what it's worth, my new year's resolution will be to do a better job at the above - and perhaps take a break from Hacker News altogether, too ;) Feel free to join the IRC channel if you want to discuss this some more. In the open, of course!

Re:Docker Image Insecurity by jsmthrowaway on Tuesday December 23, 2014 @04:09PM (#8790236)
You make some good points (and I genuinely appreciate the long and well-thought-out reply, honest), but two things stand out to me:

First, a blog post is only aggressive if you let it be. I can set up a blog and sit around shitting on Docker all day long, and in fact, the entire Hollywood ecosystem is built around that crap. There's blog posts about my current and former employers that make my stomach turn. One guy in particular has written essays about how a former employer is anti-Semitic, and they're basically all made up. My root point is just to be above it. If you want Docker to succeed, it's more effective to prove those blog posters wrong than to tell them they're wrong, you dig? Talk is cheap. "You missed this part of the announcement" is just throwing contempt around, fixing the issue is just better in every way.

The second point is I get your contempt for CoreOS and Flynn, but calling them VC-backed and swiping at them didn't help. You're VC-backed too, and you know that, so I'm left confused by your general tone on that. You're also not going to score many friends on Hacker News by contempt for the VC ecosystem, which is how that read (and apologies if I was wrong there). The us-versus-them stuff that I see you do, that included, is a pretty strong signal regarding coming to work for you, so for every asshole that won't shut up like me, imagine all the silent people reading your remarks and quietly coming to conclusions.

All in all, thank you for the reply, and I'm glad you're at least thinking about how to be better on this.

Re:Docker Image Insecurity by shykes on Tuesday December 23, 2014 @04:16PM (#8790274)
On the vc-backed point, i should have been more specific. There is nothing wrong with being vc-backed. However there has been a deliberate attempt at crafting a narrative of "docker is expanding feature scope because it is vc-backed therefore greedy an untrustworthy". That narrative is pushed by competitors who are also vc-backed, making the whole thing quite ridiculous. Yet somehow it sticks. It's very disheartening given the enormous focus on cleanly layering company success on top of project success.

Anyway, you are right. Time to focus on fixing the actual issues and ignoring the rest.

Thanks.

Re:Docker Image Insecurity by jsmthrowaway on Tuesday December 23, 2014 @04:20PM (#8790292)
No, thank you. You gained a lot of respect back just by listening and taking it to heart.

By the way, I totally get you. I'm the same way. If someone's wrong, damn my position, I want to tell them. It's damaging to a deep layer when people are wrong about your work, especially when you believe in it very strongly. I compared you to caker earlier and he and I went twenty rounds about decisive, honest response to some of the criticism. One thing he's good at, to his credit, is knowing when not to respond (usually). I'm still learning that.

Anyway, happy holidays.

Re:Docker Image Insecurity by nknighthb on Tuesday December 23, 2014 @03:19PM (#8789981)
> you remind me of the CEO of Linode

Hmm... You've just increased my respect for Solomon.

> He is equally infallible and ignores criticism.

As a long-time Linode customer, I know this to be absolutely false.

Was it your intent to so thoroughly discredit yourself?

Re:Docker Image Insecurity by jsmthrowaway on Tuesday December 23, 2014 @03:58PM (#8790181)
I was employed by Linode for three years, so I know it to be absolutely true. I'm not willing to tie my actual identity to this account very publicly (in fact, I'd rather not be here), but many people know who I am in various circles and can vouch for that. It was also a throwaway remark about handling criticism and less an indictment, because I am capable of moving on with my life and wish Linode no ill will. You're happy, so you're happy. That's all that matters.

Ignoring criticism and developing an echo chamber is dangerous, that's my point. I see shykes doing it, and caker does it too. Most of Linode has quit in the last year, too, so if you ignore the wrong criticism too long, well, people leave.

Re:Docker Image Insecurity by levlaz on Tuesday December 23, 2014 @08:51PM (#8791177)
:s/most/some
Re:Docker Image Insecurity by nknighthb on Tuesday December 23, 2014 @06:30PM (#8790803)
So you're a bitter ex-employee who comes to HN only to anonymously sling mud? That sounds about right.
Re:Docker Image Insecurity by Gigablah on Tuesday December 23, 2014 @09:03PM (#8791218)
These are some Reddit-level retorts. You're not doing yourself any favours.
Re:Docker Image Insecurity by tuffle on Tuesday December 23, 2014 @01:15PM (#8789310)
> but are not claiming that it's more secure yet

That's a bit disingenous. The CLI clearly makes a security claim here, and you know very well that users will take it as one.

I'm all for "working on ways to make image distribution more secure", but making security claims in the CLI when said security does not exist yet is -- assuming good faith -- a security bug in the user interface that should be fixed.

Re:Docker Image Insecurity by Alupis on Tuesday December 23, 2014 @01:21PM (#8789347)
> That's a bit disingenous. The CLI clearly makes a security claim here, and you know very well that users will take it as one.

This is truly one of the largest complaints about the Docker project and how they present themselves. They constantly present their product as it does X, Y, and Z today, yet upon research you will discover it really only does X, while Y and Z are planned for sometime in the future. Unfortunately, all too often X becomes some new shiny feature, while Y and Z are security related issues.

Re:Docker Image Insecurity by tempVariable on Tuesday December 23, 2014 @01:45PM (#8789497)
Docker should not show the '...Verified...' message,- I agree. I have not used it in a while. I also remember that when I used it,- I was aware that it was not verifying the images. That was my responsibility, to double check and peer review the tool which I'm using. Docker is not meant to be used by your grandma and certain level of responsibility is expected on your part.

Now, I don't care one way or another , but your message history clearly points that you troll like a maniac in each Docker thread. With all this time spent, why didn't you research a better approach, create a merge request or Docker-type of RFC post proposing a single improvement.

If you already did - great!

While I haven't contributed yet, I have read the source and saw this entry on GH: https://github.com/docker/docker/blob/master/CONTRIBUTING.md So there is no excuse not to!

Re:Docker Image Insecurity by sillysill on Thursday December 25, 2014 @02:16AM (#8795717)
I'd like you to try and get any security commits in the docker tree. Try your luck.. others have.
Re:Docker Image Insecurity by shykes on Tuesday December 23, 2014 @01:20PM (#8789345)
I agree. The message should be more clear.
Re:Docker Image Insecurity by tedivm on Tuesday December 23, 2014 @01:39PM (#8789468)
Your two messages contradict each other-

>So, we've made it pretty clear from the start

>I agree. The message should be more clear.

It seems like a bit of backpedalling here. Rather than try to blame the author for not understanding or quoting the announcement that most people wouldn't have read anyways I think you should just commit to fixing the underlying problem (first the message that's passed, then the actual feature itself) and thank the author for drawing attention to it and educating your users on the security issue at hand.

Re:Docker Image Insecurity by kstenerud on Tuesday December 23, 2014 @02:43PM (#8789810)
The first quote is in reference to the announcement. The second quote refers to the CLI messaging. Shykes is acknowledging the problem.
Re:Docker Image Insecurity by tuffle on Tuesday December 23, 2014 @01:58PM (#8789575)
It's good to hear that it will be fixed, but it's also troubling that a front-facing security issue took a #1 HN post, followed by debate in the comments before Docker recognized it.
Re:Docker Image Insecurity by shykes on Tuesday December 23, 2014 @02:23PM (#8789722)
All the issues in this post are already known and being actively discussed on the mailing lists and repo (including with the blog post author).

However nobody so far had noted that the "image verified" message is misleading and should be clarified. I'm simply acknowledging that we should fix that, too.

Re:Docker Image Insecurity by micah_chatt on Tuesday December 23, 2014 @01:04PM (#8789248)
OK, honest question: why have the "The image you are pulling has been verified" message without also printing a disclaimer on `docker pull`?

Secondly, It seems that the author did not follow the stated docker procedure for security vulnerabilities (https://www.docker.com/resources/security/). In my view this taints the author's position of revealing a vulnerability to the community and reduces it to mere attention-getting.

Re:Docker Image Insecurity by gtirloni on Tuesday December 23, 2014 @02:00PM (#8789591)
https://securityblog.redhat.com/2014/12/18/before-you-initia...

I think RH has also posted this just for the attention.

Re:Docker Image Insecurity by ewindisch on Tuesday December 23, 2014 @02:43PM (#8789814)
I really advise caution with using 'docker load'. It's more susceptible to malicious input than 'docker pull' is. That said, it has a stronger trust story. Lets not confuse the two. If you know without a doubt that your image and all of its layers are safe, then you can use GPG to do image signing and verifications, then load the image into docker.

That assumes that what you're signing and verifying is safe, i.e. non-malicious. Again, 'docker load' is less protected against malicious inputs, so under no circumstance is it safer to load arbitrary, untrusted content through this mechanism.

Finally, an interesting middle-ground is to containerize the 'docker pull', then use Docker itself to generate sanitized input to 'docker load'. It's not perfect, there are still ways to attack it, but I did put together a PoC of this:

  $ docker run ewindisch/docker-pull ubuntu | docker load
Re:Docker Image Insecurity by heavenlyhash on Tuesday December 23, 2014 @02:59PM (#8789892)
> It's more susceptible to malicious input than 'docker pull' is. That said, it has a stronger trust story.

What does that mean?

Let's look at how 'docker pull' and 'docker load' compare:

- They're both loading an image into docker.

- They're both NOT checking any signatures in any meaningful way.

- All that's different about `docker pull` is that it's fetching directly from the network.

How could 'docker load' possibly be more susceptible to malicious input?

Clearly, there is never any circumstance in which it is safe to load arbitrary, untrusted content through docker. I fail to see how loading untrusted content from the network could be safer than loading untrusted content from disk.

Re:Docker Image Insecurity by ewindisch on Tuesday December 23, 2014 @03:39PM (#8790079)
It's mostly in how Docker perform the 'docker load' and 'docker pull' that results in a different security story, not so much how it ultimately extracts the files and applies them to the filesystem.

When you use 'docker pull', you're explicitly loading a specific tag and the layers associated with it.

Docker 'load' doesn't load an image and a tag as specified by the user, it loads an arbitrary number of images and tags as specified in the provided archive.

For one example of an actual vulnerability, up until Docker 1.3.3, it was possible for a 'docker load' to execute path traversal attacks based on malicious image IDs. This was largely mitigated by the 'docker pull' code and url semantics. There was still some risk from malicious registries, but again, mitigated by having a trusted registry behind TLS.

Re:Docker Image Insecurity by wayoverthere on Tuesday December 23, 2014 @04:51PM (#8790425)
Running something inside a container does very little to actually secure it. If someone can execute arbitrary code inside a container, they can use a kernel exploit to jump outside of the container. It's important to always keep in mind that containers provide resource isolation, not a security boundary.
Re:Docker Image Insecurity by ewindisch on Tuesday December 23, 2014 @05:29PM (#8790583)
Running in a container restricts what processes may do in userspace. It's possible to take away CAP_SETUID, so that even if a process could execute arbitrary code, it could not leverage a setuid binary. There are a whole set of capabilities that admins may take away from their processes that are possible with a container. Some of these, yes, actually could protect against certain kernel exploits.

However, no, containers do not entirely protect against kernel exploits. Yet, is that what we're talking about there where the alternative is to simply execve(3) a binary, and possibly setuid to a non-root uid? Running processes with a restricted capabilities set and new namespaces is generally more secure than running them without these restricted capabilities without namespaces.

So yes, actually, containers do provide security. No, it's not absolute, but it's better than the alternative if the alternative is a naive 'execve' or one of its many frontends.

Re:Docker Image Insecurity by 23david on Wednesday December 24, 2014 @01:02AM (#8791779)
The author is a developer and professional security researcher. He responds to the security disclosure issue on the github issue page:

  > @ewindisch:
  > I also remind everyone that if they feel there may be possible attacks 
  > against the current format, not to publicly discuss this on GitHub. 
  > If you do feel this way, I'll happily entertain a private discussion. 
  > While I generally prefer and encourage transparency in open source 
  > projects, we should be careful to practice responsible disclosure.

  Disclosing security vulnerabilities is a responsible thing to do. 
  As a security researcher, it is entirely my choice how/when/if I disclose
  security issues. In this case, I'm not dropping any 0-days, just pointing out
  fundamental flaws in the current system. Fixing these flaws should be an 
  open discussion, not a private one.

  As far as "responsible disclosure" goes, it is only one vulnerability 
  disclosure approach (the alternative is not "irresponsible disclosure"), 
  and there is zero consensus about it.
Source: https://github.com/docker/docker/issues/9719#issuecomment-67...

Some good reading by Bruce Schneier re:full disclosure vs responsible disclosure: https://www.schneier.com/essays/archives/2007/01/schneier_fu...

If you want to really understand the security world, I definitely recommend attending Defcon in Las Vegas at least once to meet our Cyber brethren... :-)

Companies with the resources who publicly state that they care about security should be willing and excited to at least partially sponsor employees to go. Definitely worth every penny.

Re:Docker Image Insecurity by Alupis on Tuesday December 23, 2014 @01:00PM (#8789219)
> So, we've made it pretty clear from the start that we're working on ways to make image distribution more secure, but are not claiming that it's more secure yet

That very clearly makes Docker not secure from it's inception through today currently. That was/is the point of the article.

> but are not claiming that it's more secure yet.

What does "more secure" mean... it's either not secure or is secure.

> This pipeline is performant but completely insecure. Untrusted input should not be processed before verifying its signature. Unfortunately Docker processes images three times before checksum verification is supposed to occur.

> However, despite Docker’s claims, image checksums are never actually checked.

Things like this are by design and at the core of how Docker does what it does. Docker clearly was not built with security considerations in mind.

> Docker exacerbates this situation by running xz as root. This means that if there is a single vulnerability in xz, a call to docker pull could result in the complete compromise of your entire system.

The security implications go on and on.

Re:Docker Image Insecurity by mattmcknight on Tuesday December 23, 2014 @01:05PM (#8789254)
It doesn't make Docker "not secure", it means that you need to verify image files in the way you verify other files you download. These changes are irrelevant to the security of Docker itself, for the way I use it with my own images.

"it's either not secure or is secure." Security is always a matter of degree.

Re:Docker Image Insecurity by Alupis on Tuesday December 23, 2014 @01:29PM (#8789396)
> it means that you need to verify image files in the way you verify other files you download

That is difficult to script automation that handles all cases. They have mechanisms built in that claim they provide the security you would expect, except that this article exposes their claims are absurdly false and actually dangerous because most users are not going to take the time to verify what the code is actually doing -- ie. they will just trust Docker's claim and move forward, potentially blindly exposing themselves.

> Security is always a matter of degree.

I agree, except in this case Docker has totally cast most considerations of security out the window. Simple "best practices" like verify user/untrusted input before you begin to work with it are completely ignored as evidenced in this article.

> Docker users should be aware that the code responsible for downloading images is shockingly insecure.

The issue is Docker sweeps this under the rug and continues to claim their platform is secure. Users trust this, and leave themselves exposed.

> It doesn't make Docker "not secure"

It sure seems to indicate that Docker is certainly not secure.

Re:Docker Image Insecurity by jaegerpicker on Tuesday December 23, 2014 @01:11PM (#8789286)
More secure means exactly what he says. Nothing is ever secure unless it's unplugged from the network. Security is a series of doors between your data and the attackers, making something more secure is simply adding more doors in front of it.
Re:Docker Image Insecurity by Alupis on Tuesday December 23, 2014 @01:18PM (#8789328)
> I assumed this referenced Docker’s heavily promoted image signing system and didn’t investigate further at the time. Later, while researching the cryptographic digest system that Docker tries to secure images with, I had the opportunity to explore further. What I found was a total systemic failure of all logic related to image security.

The issue is Docker has never taken security seriously. From inception up to today. Security is an afterthought, very clearly evidenced by shykes comments above, and by the code being committed.

Security for such a core piece of infrastructure, something Docker Inc hopes to thrust into all servers and devices someday, should rightfully be the first and foremost thoughts and part of the initial design of new features. Instead things are designed and developed, and security slips to the side. It takes articles like this to actually motivate Docker to do anything about it.

Re:Docker Image Insecurity by gitdude on Tuesday December 23, 2014 @05:54PM (#8790677)
Professional Advice for shykes: Do some background research + internal prep before answering/commenting here on HN. You represent the Docker brand and over time such hastiness will erode Docker's credibility. Best to be silent than defend every incredulous comment about Docker on HN.