That's because security requires defense in depth. If the failure of a single security control can invalidate your security model, your security model is inadequate.

It should require multiple things to go wrong for catastrophic failure. This is a lesson from engineering that hasn't made its way to software development yet (outside of security engineering, anyway).

I believe several terms here need a context, without which they are meaningless. For example, what is an acceptable security model to download and install an OS, how do you exactly define "defense in depth" for the act of downloading and installing an OS, etc.. That will help us define what controls should be in place for the overall OS-download-and-install experience to be secure.

Defense in depth is just an industry term for redundant security. For example, you can mitigate tampering with data transfers by signing the data itself AND ALSO by signing the channel it's transferred over with TLS. If a flaw is found in one of those methods, the other will still protect you.

The process of listing all the security failure points and documenting the redundant mechanisms to protect them is called threat modeling.

For a system that installs OS-level binaries as root, it would absolutely be appropriate to threat model it and hold it to a defense in depth standard. In defense systems, they often require three levels of defense in depth, the last being an air gap network.

Yes, this works. In theory. In practice, a simple model is needed that everyone can follow and implement consistently. That, does not exist.

Lookup "threat modeling" and you will see how abstract a notion it is (even your comment calls for a "redundant mechanism" that may not be exactly what you are looking for), and how little information is available. End result? Most do it for the "checkbox effect". Don't get me wrong, I am not trying to obliterate what you said, just putting some factual data around it.

You're right that it's not simple. In fact, studying security-senstivie ways in which software tends to fail and how those failures can be mitigated is an entire field unto itself. Software developers can't be expected to get it right on their own. That's why all major software companies have security engineers on staff.

Open source projects, unfortunately, rarely have such contributors. Probably because building stuff is more fun than threat modeling (which can be quite tedious to do properly).

One really obvious thing that occurs to me is that apt uses separate processes for network I/O but fails to reduce their privileges.

> If the failure of a single security control can invalidate your security model, your security model is inadequate.

As an example, when an admin gets an AWS Security Group wrong, thereby exposing database servers / redis / customer data. Consequence... multimillion $ fines, brand/reputation damage.

It's kind of sad how badly things are set up to fail sometimes. :(

OTOH, they would have been right if there had been (yet another) bug in openssl/whatever lib would be used for https.

FWIW: 16 vulns in apt in NVD [0]; but 202 for openssl [1]

[0] https://nvd.nist.gov/vuln/search/results?form_type=Advanced&...

[1] https://nvd.nist.gov/vuln/search/results?form_type=Advanced&...

APT already supports HTTPS. Enforcing it by default wouldn't increase APT's attack surface significantly.

It would decrease the number/quantity/capacity of available mirrors. I don't know if that quantity would be significant.

You would lose the ability to do transparent caching which I agree is rather annoying, but I think most environments where that sort of caching occurs (mostly corporate and school networks) also have the means to explicitly configure client machines to use an internal caching mirror.

Those environments tend to MITM https traffic as well. At least the companies I've worked for can.

Those companies really need to learn what privacy means.

You could run a caching mirror for such things, such as artifactory for example.

OpenSSL is not the only available implementation of SSL.

Realistically, it's the one that's gonna be used in apt though. (E.g. the existing HTTPS transport uses curl, which uses OpenSSL)

libcurl has many TLS backends. APT before 1.6 actually uses the GnuTLS one.

APT ≥ 1.6 doesn't use libcurl; it uses GnuTLS directly.

I stand corrected, thx.

Sure, my point was more that if it's a serious concern to use OpenSSL (Personally I don't think it is, as a lot of work has been done in recent years to improve that codebase), that other options are available.

So your argument is that bugs in OpenSSL (a necessarily very complex piece of software) mean that using SSL to increase network security is a bad thing?

No, their argument is that bugs in OpenSSL mean that using SSL when it does not actually increase network security is a bad thing. Reduce your attack surface. Use the things you need to use; don't use things you don't need to use.

Whether apt using OpenSSL would in fact increase network security is a separate and debatable question, but the argument as stated assumes it would not, and is sound.

Defense in depth is a thing.

SSL provides some security guarantees.

Using signed package databases also provide some security guarantees.

Both may overlap in what security they provide.

BUT!

If one fails, the other can continue to provide a subset of the previously available guarantees.

Not in the case of OpenSSL, no. (Some) OpenSSL issues, just as (some) apt issues, end with RCE. Game over.

Priv-sep, correctly handling untrusted files (e.g. 1. check signature, then 2. execute whatever; not the other way round), memory-safe languages, etc. would be more welcome additions.

> Priv-sep

Apt even has the had part already implemented by separating the network I/O in other processes. Only problem is that those currently write directly to system directories, but that can be fixed.

And overcoming both is much harder than overcoming either on their own.

In the worst case, you only need to overcome one. And you ~double your attack surface.

Which one? If you gain control of a server, you'd still have to overcome signing... and you only control that single server, not all replicas (for a bug like in TFA).

Could you describe a way to have double the attack surface that would effect the majority of peer servers?

No. My argument is that both arguments do exist and security is all about middle-ground. Do we add (yet another) layer of (bug-riddled) software to defeat one possible sort of exploits, or not? How much does it cost? (Time, money, etc.)

Other options include: not handle http directly in the package manager but use a known-good library (curl?); do priv-sep; not having the package manager execute code from a file before it checked its authenticity...

My personal opinion on the topic is that authenticating servers is a good thing to do (https does this; encryption is an added benefit), but time has shown that https is broken: libs are full of holes; the CA model is broken by design. Maybe share updates using ssh?

Oh for god’s sake will you stop spreading nonsense FUD about https? https is not “broken”. The majority of websites run on https now; most system update mechanisms with the VERY notable exception of apt use https to serve updates.

Sure, there are bugs in libraries, but seeing as https is already widespread you’re not exposing yourself to MORE risk by using https over plain http, and you mitigate attacks like this post. Any random coffee shop, untrusted public WiFi, or attacker with a Pineapple could have used this attack to MITM HTTP apt, whereas the attacker would have to compromise an upstream mirror to pull off the same attack over HTTPS.

And re: the CA model, if you’re THAT worried about compromised or fake certs, then pin the cert for a root server like debian.org, then download PGP-signed cert bundles for mirrors and enforce certificate pinning using those bundles only. Done. Apple and Microsoft use cert pinning for their update systems (IIRC).

CA-model requires "extra" investment, that not many are willing to make. Plus there is a secure way to distribute an OS, which does not require https [download over http, obtain signed check-sums from a trusted source, verify, and proceed with installation].

Letsencrypt.org.

Stop it, just stop it.

> The majority of websites run on https now

We are talking about a specific use case of https: software repositories, which are far higher-value targets than your random website, with another set of challenges. Your package manager can actually do some things as root; once it's owned, your system is Game Over.

Adding yet another lib on top of the (?) most important piece of software on your computer is not a risk to take lightly. There are more elegant solutions (signatures, priv-sep, not trusting anything until authenticated, etc.) that require less risky code to run, and fewer people to come into play.

> Sure, there are bugs in libraries, but seeing as https is already widespread you’re not exposing yourself to MORE risk by using https over plain http

Irrelevant. We're talking about instant game over if it goes to sh.. even if just once. More attack surface = more vulnerabilities.

If you're that paranoid about OpenSSL, then just sandbox it. Throw the entire `apt-transport-https` subprocess in an unprivileged context. Done.

With https libs you're trading one potential issue with another class of issues. Https implementation may have an rce but (this happens extremely infrequently and can be patched quickly). At the same time, it prevents the whole class of issues of MitM, whatever element of the underlying system they would target. (Which potentially need client changes to multiple elements to get fixed) This is a pretty easy decision to make.

I think the point is that adding https over http for the current system would always improve security. At it's most broken, https is at least as secure as http and therefore wouldn't reduce the security of the overall system. It adds one more hurdle for an attacker to clear.

Similarly, the apt team ignoring a bug like this "because it's protected by https anyway." Is an invalid argument.

> adding https over http for the current system would always improve security

No.

If an attacker can inject packets that break your SSL lib, but wouldn't have broken your package manager, you added a vuln.

Example: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2003-0545

Fair. There are specific library attacks that could result in RCE. However that is also true against curl, ssh, and for that matter could be introduced into your http library at some point. The question then becomes what library do you trust most? OpenSSL is attacked and tested constantly. Things have been found (in your example in 2003!). They have been fixed. Apt can choose to stand on it's shoulders, or go through the entire process themselves by putting together a patchwork of their own solutions that will no doubt get less testing by whitehats and be a juicy target for blackhats.

Even ignoring the fact that there are far better libraries than OpenSSL today (eg: BoringSSL), apt already implements a sandbox-like approach (as the article explains); I'm not sure if the subprocesses are actually sandboxed, but obviously they should and at that point, a vulnerability like the one you cited shouldn't let the attacker escape the very narrow sandbox.

That, and Debian would fuck up the implementation.

Lest anyone forgets:

https://www.schneier.com/blog/archives/2008/05/random_number...

How many of these would result in RCE?

Hard to give an exact amount but there have been loads of potentially exploitable memory corruption issues.

64?

https://nvd.nist.gov/vuln/search/results?form_type=Advanced&...

Fulltext search for "rce", which finds "resou_rce_", "sou_rce_", does not give a number of RCE vulnerabilities.

Except that "EXACT MATCH" is enabled, try yourself.

(It should be noted that it /does/ match on "possible RCE", which buffer overflows are often tagged with.)

I have. Search for "ontgome" and it finds the bugs containing "Montgomery" (I have taken your url and just replaced the search word):

https://nvd.nist.gov/vuln/search/results?form_type=Advanced&...

I'm not saying none of the results from your search are RCEs, but not all are, and many are fairly speculative.

Argh, that's frustrating, I checked 4 of them and thought it was fine.

The problem is that there seem to be many classifications of remote code execution including buffer overflow and "code injection" and you can't choose multiple. :(

Yes, I also was surprised that the search didn't have more useful tools (e.g. search by high ranking in individual factors, or even just sort by severity: confirmed RCEs should all by very high)

As far as I can tell, NIST doesn't directly directly use the term RCE. CVE-2010-5298 is an OpenSSL vulnerability that allows data injection that could potentially result in code execution, but there's no easy way to see that from NIST's categorization:

https://nvd.nist.gov/vuln/detail/CVE-2010-5298

that's an additional problem with relying on text search.

Makes me chuckle, since my comment on the HN post yesterday that highlighted that someone will be bitten if security principles are ignored got downvoted.

I was laughing as well, software people (the HN and open source community in particular) are so stupidly opinionated sometimes, it makes me just cringe whenever its impossible to ignore their dribble.

There really is no excuse not to use HTTPS in 2019, period.

I will never understand why people are so proud to have gotten rid of "needless" security. It always ends the same way...

I think I understand the exploit but I don't understand whether apt using https would prevent it or not. The author says:

> Yes, a malicious mirror could still exploit a bug like this, even with https.

and:

> I wouldn’t have been able to exploit the Dockerfile at the top of this post if the default package servers had been using https.

So which is it?

Essentially adding HTTPS would make the attack harder to exploit. It's not that HTTPS is a pancea (it's not) but that it raises the bar to a successful attack.

With HTTP, this can be exploited by anyone who can MITM a connection between you and the APT server or has control of your DNS.

If you consider all the cases like wi-fi hotspots, that's (potentially) quite a large set of attackers, and a relatively easy attack to pull off in a lot of cases.

With HTTPS, the attacker has either to compromise the whole APT mirror or has to get a valid HTTPS certificate for an APT mirror. This is likely harder to pull off, especially when you look at the work on improving CA security that the browser vendors have been doing over the last couple of years.

We're talking about a million dollar software designed for governments and is sold only to highest bidders. I refuse to believe using HTTPS would be helpful here. This attack uses state-of-art to exploit HTTP and there is no reason to assume it wouldn't use state-of-art if it were HTTPS.

HTTPS: A malicious mirror operator can pwn you.

HTTP: Everyone can pwn you.

Not saying the first one is ideal, but the second one is definitely worse.

With HTTP an attacker still has to MITM the connection between you and the mirror operator. So, definitely not "everyone".

That includes: coffee shops, ISPs, employers, everyone who can hack their routers, anyone who can spoof DNS, etc. That might as well be "everyone".

STOP IT. Though shall use HTTPS.

Fair enough. I agree that HTTPS is valuable here. I was just being overly pedantic, my bad.

the moment we are talking about monitoring user's software base, we are practically already talking attackers at the skill level of nation states, so yeah "everyone" in the subset of plausible attackers.

Both. Without HTTPS, you can execute the attack if you can MITM the connection to the package repository. If HTTPS is used, you need to be the package repository to do the attack, or need a certificate to MITM the connection so you can pretend to be it.

With HTTPS a MitM attacker can still refuse to serve a specific package.

https prevents an attacker which can compromise the network. It does not prevent an attacker who can compromise the mirror. The author can't compromise an existing mirror, so they wouldn't able to exploit it (through the network) if the servers were using https.

Every time this site comes up people entirely miss the point in this regard -- Debian operates a large voluntary network of mirrors. You are not trusting content coming from Debian, you're trusting it coming from the mirror. SSL only secures the link between the client and the potentially compromised mirror, it does not solve problems like the one from the article.

Meanwhile it's worth pointing out that OpenSSL has historically been one of the buggiest pieces of code in existence. Despite this being a game over RCE, it's the first of its kind in many years. If OpenSSL had been in the mix, Apt would have required forced upgrades /far/ more often. https://www.openssl.org/news/vulnerabilities.html

I don't think that an argument that using HTTPS actively decreases the security of a connection really holds all that well.

If you don't think OpenSSL is a high enough quality implementation, there are many others to choose from.

Even with a range of mirrors, it would still raise the bar for attackers, to require HTTPS.

He never explains why he wants to use HTTP, it’s only about why he thinks HTTPS isn’t nesscary.

HTTP is the null hypothesis, since it's simpler. Usually there is a great reason to reject this null hypothesis - it prevents security vulnerabilities. But if there is no added value, then there is no reason to do it.

Consider, why not double-wrap your stream? Put TLS on top of TLS on top of HTTP?

It's worth noting that a large number of people don't agree with you that HTTP is the null hypothesis. Instead, they think that HTTPS is a security/privacy best practice and a great part of defense in depth.

You can see this pro-HTTPS opinion all over this discussion.

As for your "consider", I personally do double-wrap many streams: I have a VPN for my browser. The VPN is great for hiding my home traffic from being spied on by my ISP. Without the VPN, HTTPS streams would reveal hostnames (SNI) and IP addresses to my ISP.

> Consider, why not double-wrap your stream? Put TLS on top of TLS on top of HTTP?

If it's the exact same implementation then that doesn't really add a second layer. If, however, I am provided the option to run HTTPS over a VPN tunnel, then I would happily do that in a heartbeat. In fact, I frequently do run my web traffic over a proxy, thereby giving it at least two layers of encryption.

Yet it’s actually not simpler for the user, since their transfer can then be tampered with either by accident or intentionally leaving the user with a broken download and then what do they do? A redownlaod from a different mirror makes no difference.

The situation you described is the same thing that happens with a MITM attack with HTTPS. You would get a failed download from any mirror.

Do you have a response to my question? "Consider, why not double-wrap your stream? Put TLS on top of TLS on top of HTTP?"

It’s not the same, Comcast and other ISPs don’t tamper with HTTPS, and if they break the HTTPS connection then it’s a clearer problem for the ISP to troubleshoot than corruption.

Sorry I don’t understand what double wrapping has to do with it, or why you’d ever do that.

Because that just makes things slower for no good reason?

Sounds like an argument for rejecting HTTP+TLS single-wrap too. (For apt — not in general.)

I was being glib because I didn’t think I needed to explain fully, but here we go.

Double-encrypting something with the same technique is pretty much always a sign of cargo cult crypto. Modern ciphers, like those used by TLS, are strong enough that there’s no reasonable way to break them applied once, and the downside is that applying them twice is making things slower than they need to be for zero added benefit.

On the other hand, TLS and PGP are very different things serving very different purposes, so nesting those makes sense. There is an added benefit from TLS, namely that you ensure that everything is protected in transit - including the HTTP protocol itself (which is currently not protected and which might be subject to manipulation as shown in this post). Plus, it provides some resistance to eavesdropping (and with eSNI + mirrors hosted on shared hosts, that resistance should improve further).

also, the apt way to fix this would be to a) move release.gpg out of the package path and b) require the release.gpg to be wrapped and signed with the previous valid key instead of being accepted blindly

Some country's firewalls distrupt https, which makes downloading things via https difficult.

So you create an non default http mirror for that minority, instead of making the majority insecure.

so if north korea is subjugating some poor souls over there, the whole world must suffer along? there could be a setting with a big warning to disable the default HTTPS behaviour...

Which countries? I’ve only seen HTTP connections tampered with in practice, and China’s GFW blocks HTTP no different than HTTPS from what I’ve seen.

Also some companies, to allow IDS to inspect traffic without having to extract keys from clients.

They (ubuntu, at least) have always been overtly hostile about this issue, which seems strange. I recall seeing open issues that just devolved into argument for many years on this. I don't get it.

Discussed yesterday:

https://news.ycombinator.com/item?id=18958679

I was just coming here to post that

You know that HTTPS has the Location header too, right? This attack works just as well against HTTPS clients.