Hacking aside, Docker is an invitation to trouble. Anybody can publish a binary blob, and users are expected to blindly trust it. It's centralized. It doesn't have a context of "trustworthiness" yet I don't recall docker ever warning me that the image I'm downloading could have been the work of any person.

Shortcuts all around -- kind of reminds me of MongoDB. Sad it's the primary player...

> Docker is an invitation to trouble. Anybody can publish a binary blob, and users are expected to blindly trust it

How is this issue specific to Docker? Anyone can download a random library off github, use a shady linux distribution, or install utility tools loaded with spyware.

I don't think Docker aims to solve issues relating to trusting upstream software. It's a tool to help package applications, just like how tar allows you to package files. What you put in it is up to you.

That's a strawman because the shriekingly-obvious difference is ease of facilitation. With Docker, it's a mere couple of commands to pull a box and run it. Docker weakens trust because it lets anonymous people, as well as trusted ones, upload images that can be immediately run... but without a proper chain-of-custody, QA or assurance that an image hasn't been manipulated on Docker's side. It's spray-and-pray DevOps. Image integrity has to be solved on Docker's side with end-to-end integrity or it's all for naught... this is something that cannot be solved within a container or separate from Docker, it must be universal, mandatory and trustworthy.

> Docker weakens trust because it lets anonymous people ... upload images that can be immediately run... but without a proper chain-of-custody, QA or assurance that an image hasn't been manipulated

How is this github any different?

Many package managers that support git as source allow to pin to a specific commit sha. That's as far as I can see a quite secure way to keep using an uncompromised/verified version. It's not the most popular feature but people do it every now and then, probably it should be done more.

I wonder if docker allows this and on the other hand if that's even feasible for say application images, given that applications must be updated a lot for security reasons. Of course if the Dockerfile's parent reference is not pinned, that does only help to some degree...

You can pull an image using the sha:

docker pull ubuntu@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2

Which effectively nobody does. Package managers and distribution packaging systems default to the safe method rather then defaulting to insecure rewritable tags.

To be fair, the docker.io/library/* images are signed but no other images are and there are a bunch of issues with how the signing policies work for users that want to enforce that some images must be signed.

The important thing is that tags are signed and up-to-date, like how git tags work or how Debian signs its entire repository as a unit (via the Release file) rather than having developers just sign individual packages. Otherwise, even if it's signed, it's subject to downgrade attacks.

Installing known-vulnerable old versions of legitimate software can be just as bad as installing custom malware.

Sure, that's how almost all package managers work. I can't think of a modern package manager from an "enterprise" distribution that didn't have a lot of the features of TUF[+].

And as I said, only official-library Docker images are signed. All other images are unsigned and even for third-party repos you can't force Docker to verify all images from a given repo (you have to enable it globally, which breaks the utility of a local "docker build").

[+] Arch is the only counterexample I can think of and I'm not even sure if my memory is correct.

I do it! Everything I pull is pinned with sha256 since I use Nix/Kubenix, so I'm required to pin sha256 if I'm fetching from the Docker registry (or build the package deterministically myself.)

The way image signing works with Docker is that there is a signature tying a tag to a sha256. If you use the sha256 directly you get immutable sources, but now your source isn't signed anymore -- how are you sure the hash is correct?

It's a bit of a pain, you need to build, push, pull, then get the sha. I suspect it would be done more if there was actually a decent UX for it.

> How is this issue specific to Docker? Anyone can download a random library off github, use a shady linux distribution

Strawman. Anyone can use Debian Stable or at least Testing.

Libraries off Github literally have the source available for you and the community at large to vet. And you'll find almost no sane shop on the planet where people are allowed, hell encouraged to use shady distros or install random utility tools in production the way they are encouraged to pull unchecked binary blobs from Docker Hub in an often non-reproducible manner.

> Libraries off Github literally have the source available for you and the community at large to vet.

Nobody read the source code for this exact reason: “the community is here to read it so I won’t".

Well, for most libraries used in Desktop Linux, a significant number of stakeholders (developers+users) exist, which actually care for development and the complete thing itself. Also the libraries generally are designed for solving problems and not getting github-stars by bots/dependency-building.

For docker (and npm for all that matters) _a lot_ of important dependencies are basically simple one-off "developments" with a single developer and no userbase at all caring for them, because they don't really solve any consistent problem, being basically just created to increase the visibility of its creator on primitive metrics. The community is there for high-level packages, but the dependencies lurk in test-scripts and seldom-used functions carefully placed by some idiotic digital nomads for their personal CV-polishment (ehm, not looking at you: https://github.com/sindresorhus/shebang-regex). Have a look at where this package is used (basically only in cross-spawn, where there are 10 other similar dependencies), then think about, how much effort creating the dependency hierarchy was, then look up who contributed the changes, where this micro-package was required and finally decide whether this was some thing sane people would do or if it's just for personal gain...

> Nobody read the source code

In Debian we review and vet packages.

Speak for yourself, please - there are plenty of places that aren't into cowboy coding.

> Libraries off Github literally have the source available for you and the community at large to vet.

If Github was compromised, it would be easy and obvious to insert malicious code in a repository, but hide those changes from anyone on the github website.

Which you can avoid by forking the mainline repo and depending on your fork.

Images on Docker Hub don't even need to share their Dockerfile, to talk of all the source/etc that went into their build.

>> If Github was compromised, it would be easy and obvious to insert malicious code in a repository

> Which you can avoid by forking the mainline repo and depending on your fork.

If github was compromised, the it would be pretty easy to generate forks with the same compromised code.

> It's centralized

Actually only short names go to docker hub, one can setup their own registry and use it via dns names.

Example: docker pull quay.io/letsencrypt/letsencrypt

Exactly this. For the docker images we use in production, we fork the corresponding git repo, build our own image and push it to our own local docker registry and pull it from there. It's fairly easy to setup in fact.

Out of curiosity do you resolve it so that the image is FROM scratch or do you rely on alpine/some other base image?

I forked an ubuntu image and then used it as a base for all my projects. It doesn't come for free though, you will need to periodically run security updates and then rebuild all images that depend on it.

Not docker, but library hosting, in general. My company maintains client libraries for 6 languages, also hosted in the popular place for that language. The standards for account management and authenticating the libraries are all different. Some have scary-little security, some have painful security.

Agreed 100%. It's insane the practices that fly in our industry. It's as if we didn't know any better.

One day, there's going to be a colossal compromise, and that might finally change where we place security in the priority chain.

But it was possible to see this trouble coming from way back. Docker, Inc. took on over $150m of VC investment up to the end of 2015. One hundred and fifty million dollars. How do you possibly plan to show a return on such an investment? The only way is to get one of your "services" injected into peoples pipelines as a critical component, no matter how questionable the fundamental necessity of that service is. But of course, you own the tool, you get to design the workflow and do your best to shape your users worldview.

I do wish developers would be a little wiser around these things, especially when they see companies taking such huge amounts of capital. I found it quite depressing to watch the unquestioning way development communities assimilated the docker worldview.

I was originally going to argue with it being "just a matter of time" -- there is such a thing as good security practices. It's certainly not "just a matter of time" before Microsoft or Google see such compromises. I'm pretty confident that these companies have their sh*t in order.

But no, not Docker.

You're totally right; with as important as their registry is to well funded attackers, and as startup-y and "agile" as they are, and as godawful as the security practices are that underlie their tools and standards... they hadn't a chance. They still don't.

There is no reason to expect them to get better.

Fun fact, there was a universal XSS vulnerability on google (including search, support, accounts, cloud, etc) found just last week [0]. I'd say it's always just a matter of time. That doesn't mean they don't have everything in order, but securing everything as much as possible is half the battle. The other half is a solid response when things do happen, which we will now see in how Docker handles this situation.

[0] https://twitter.com/WHHackersBR/status/1118393568656334850

And do we ever find out how much that was being exploited "in the wild"?

You'd want the XSS vulnerability to be on accounts.google.com. Much more to do before you can successfully exploit it. You still need to get people to come to your malicious page that exploits it. Then it's the question if your attack won't show up on Google's radars for abnormal behavior. Most likely for Google's security - since their landscape is so big - XSS vulnerabilities are considered a given. Then as soon as abnormal behavior is detected Google gets to discover the XSS vulnerability.

we grit our teeth and "believe" that anyone traceably affected got an email directly from the company or something :D

(that said, google main page vulnerable to xss is kind of like... what, we're afraid someone will take over google and put some cryptominers on the google.com main page?)

Well, a compromised google.com main page could return malicious search results for certain queries. How many Windows sysadmins install PuTTY by googling "putty", and then installing an executable from whatever site shows up in the first couple of results?...

If the primary install method is "search and download whatever manually from the internet," you have bigger issues than a potential Google compromise: create a site with better ranking than the canonical HTTP (!) download page, MITM the HTTP download, whatever.

The Microsoft Approach... 'people totally didn't access your email body... except we eventually owned up to it after it got leaked'

Where did they deny that anybody's email bodies were read? I'm looking for it and I can't find it. I only see that they told the other 94%(?) of people that unauthorized access did not reveal the contents of their messages in particular, which seems to be truthful?

Initial email said the body wasn't affected, and motherboard asked for a confirmation, so they said 'Yes'.

6% of the people received a specific email saying the body of their email was accessed and they had to backtrack.

Well the email said:

> This unauthorized access could have allowed unauthorized parties to access and/or view information related to your email account (such as your e-mail address, folder names, the subject lines of e-mails, and the names of other e-mail addresses you communicate with), but not the content of any e-mails or attachments, between January 1st 2019 and March 28th 2019.

Notice it says your email account. The whole email is about the account of the recipient, not those of other recipients. Given that they explicitly worded it this way and people clearly misinterpreted it to mean something else, I hope you can forgive me for being a little skeptical of third-party anecdotes that suggest Microsoft claimed nobody's email contents were accessed...

“We are enhancing our overall security processes and reviewing our policies. Additional monitoring tools are now in place.”

Why wasn’t that the case before?!

Because "Security is a journey, not a destination".

There's always a way to enhance your processes, monitor more indicators, etc. or otherwise improve your security.

Sometimes you don't know what to monitor until you know the attack vector. We are only human

1. Because humans aren’t perfect. 2. Because mistakes happen. 3. Because there’s a cost to everything: if you want better security, it’s going to cost you more, immediately. And we don’t always estimate trade-offs correctly (see points 1 and 2).

Exactly.. life in general is about constant refinement.. if today's hacker could time travel to 1999, she would be in a nirvana of Bind, SSHv1, Apache, IIS etc.. vulnerabilities. Hacks happen and we learn and improve, even down to the language being used.. a la Rust.

You are aware that Google identified a vulnerability so awful that they hid it from the public so as not to draw government scrutiny, did not retain access logs, and ultimately shut down a major public application?

It wasn't authentication credentials, but still.

Which vulnerability was this?

Presumably the Google+ exfiltration issue.

> The bigger problem for Google isn’t the crime, but the cover-up. The vulnerability was fixed in March, but Google didn’t come clean until seven months later when The Wall Street Journal got hold of some of the memos discussing the bug. The company seems to know it messed up — why else nuke an entire social network off the map? — but there’s real confusion about exactly what went wrong and when, a confusion that plays into deeper issues in how tech deals with this kind of privacy slip.

(https://www.theverge.com/2018/10/9/17957312/google-plus-vuln...)

it is of course just a matter of time for either of the companies you mentioned to "be hacked" (obviously it's happened countless times with Microsoft, both the OS and their cloud services like O365, and there was a recent high profile revelation that the google apps suite APIs exposed user info to developers). the difference is incident response and layered security.

as long as you're using software somewhere in the stack that isn't like maturity level 5, AND you don't have constant audits looking for novel attacks on working-as-intended systems, you're pretty much guaranteed to inherit (or create) a vulnerability at some point, and if you're important enough it will get exploited. the reason that doesn't mean we should start modeling computer systems as "living organisms that eventually get old and die" and should keep modeling security like war is that when you get hit, you can respond. all the layers matter, and insofar as Microsoft or Google do it right, they primarily do it right by having a mature process for monitoring, patching, isolating, etc.

as for docker hub though, yeah i'm totally with you. i'm just saying we shouldn't overestimate the preventive capacity of anyone, honestly. if you're doing anything important over the internet at all, you're making some compromises somewhere.

here are 2 links to things i handwaved at above, for example's sake:

https://www.wired.com/story/microsoft-email-hack-outlook-hot...

https://www.forbes.com/sites/kateoflahertyuk/2018/10/09/goog...

It's made worse by the fact that only a few major images are used as bases. That's normally good for security, as they are highly vetted and quickly updated, but if they could be compromised, say Alpine or Ubuntu Clould, even for a minute, countless images would be built using the compromised base and it would be very hard to ensure they were all rebuilt.

As I understand it, there's no element of signing from the actual devs of an image, just from the central trust service of Docker Hub.

I don’t think Docker has a way to revoke individual image hashes. Or does it?

That doesn't mean there aren't plenty of things they could have done to make this more secure. The fact that you can just `docker login` with the same credentials that allow access to your entire registry is pretty poor security design IMO.

Is hacking even needed?

There already have been questionable images hosted there ... just by users uploading compromised images. No hacking needed.

There's a difference between alpineworm:latest and alpine:latest. Someone would have to choose to download the questionable image, while someone compromising a base image could go unnoticed for quite some time and have a massive install base since it's used in so many other images.

This raises the question of whether any high profile images were targetted by the infiltrators?

Do they offer end to end encryption and signing. That would make your ci say f-off it the image has been played around with, and also protect any secrets, although there is no need to have source code or passwords in the image anyway.