I've always been very trusting of open source software. I've often thought "surely someone else has looked through this source code" and just assumed that malicious code never hits stable releases. The same goes for "verified" binaries, packages, etc... apparently that is not always the case.

What are the possibilities that such code could make its way into some piece of extremely popular public facing software, say Apache? How many cleverly hidden "bugs" already exist that open us up to complete pwnage for the clever bastard behind them?

Just think about the havoc someone could do with a few popular but nasty PyPi or RubyGem packages...

Did you read the link as I did? The server hosting the packages was compromised. This could have happened to any software, not just open source, and I'm sure it has.

>The same goes for "verified" binaries, packages, etc... apparently that is not always the case.

If proper package signing was in place, like most modern distros support, then this couldn't have happened. The backdoor was not in the main code in SVN or in Git, which specifically protects against this exact problem, but added to unprotected releases dumped on a server.

>What are the possibilities that such code could make its way into some piece of extremely popular public facing software, say Apache?

No bad code was added to the code base.

> The backdoor was not in the main code in SVN or in Git, which specifically protects against this exact problem

Pedant alert, but Subversion doesn't protect against malicious manipulation; it doesn't checksum its commits.

I think that phrase was intended to relate to just Git, although I could be wrong.

Git uses a lot of hashes but they're for speed, not for security. If you want security from Git, you must cryptographically sign your commits, which is pretty common in big projects.

I was specifically referring to Git.

While those points all matter, don't think of this as an open source issue. This can happen to any software provider, and often does.

the back door wasn't ever in the version control, it seems, which means it was probably someone broke into the servers storing the code after it was packaged. code is definitely read by someone, but open source projects never have the resources to continuously audit ever single package they have for download.

on the contrary, many open source projects sign and hash their packages, to anyone that verifies the signature and hash, the tampering will be immediately apparent.

Anyone who can backdoor a release on their FTP server can easily upload a new CHECKSUM file. Many users probably do not bother to check the hash. Even if you use public key crypto to sign the hash, users probably don't know your public key in advance. An attacker can sign their malicious hash with any valid public key they like.

yes, and its turtles all the way down, but its much harder to compromise (multiple) key servers than a simple download mirror. Its also trivial to setup some sort of totally unknown checking setup, that downloads the packages daily and checks if they are properly signed, and activates big red sirens if it is not.

MD5 hashing is known weak, and must not be used for security.

MD5 hashing is currently used to give quick and simple check on transmission download errors, rather than malicious attacks.

That's why you should use SHA-hashes when you need cryptographic security.

Typically the people looking "through the source code" are the developers working on the the project. Both occasional contributors who want to fix a bug or add a feature for themselves, or the main developers taking those patches and adding them to the main release branch after reviewing the code.

All those developers use VCS repositories, not the releases distributed through FTP sites and such -- which were the files affected here.

Wasn't there a serious backdoor situation with some Linux distros (Ubuntu?) last year? Some guy came out detailing a 10 year contract of silence or something...

You mean the OpenBSD IPSec backdoor controversy? Nothing found AFAIK.

HN discussion at the time: http://news.ycombinator.com/item?id=2029175

As I recall, numerous developers and security experts tore through the relevant code there, and they didn't find anything.

security is not a matter of being open or closed source

"given enough eyeballs, all bugs are shallow" There aren't enough eyeballs.

The comments on this page immediately reminded me of Ken Thompson's point: "You can't trust code that you did not totally create yourself."

He wrote this right after demonstrating how to create, step-by-step, an undetectable trojan horse in the C compiler. Here are the steps for creating such a trojan horse:

http://cm.bell-labs.com/who/ken/trust.html

[ Also see http://news.ycombinator.com/item?id=2642486 ]

You can't totally trust your own code, either. Look at all the buffer overrun vulnerabilities that have been found over the years.

Thanks for sharing the link. It's one of the finest things about programming I've read. I enjoyed it a lot.

So several Horde releases were trojaned for three months? That's pretty terrible. Good on them for coming clean.

What's the best way for open source projects to make it easy for their customers to get verified downloads? A lot of packages post MD5 checksums but no one tests them when downloading manually, do they? Automated signature checking on Debian packages seem to work better in practice; homebrew also verifies download checksums automatically.

Checksums provide no security.

Idea: protocol- and package-format-independent verification of packages, with trust rooted in DNSSEC of the ultimate source domain using DANE.

That's not entirely true. Since i found how to poison my ISP's PeerApp "invisible" cache servers i started to check MD5 when downloading manually. It does cache big files but not small ones. Here is the link for technical details if you are interested. http://godlessmechanics.blogspot.com/2011/12/tale-of-sneaky-...

A PGP signature would work, but you would:

1. Have to control the signature that's provided 2. Have to have a trust system in place so that people realized that the signature provided wasn't legit.

I'm not sure the personal public key infrastructure (web of trust) is developed enough to absorb that.

All the most used Linux distros use a PGP-based check before installing packages.

These distros are distributed with a keychain that contains keys for all the approved developers/uploaders. What you have to trust is the fact that the keychain you have is the real one, but that is easy to do. Once you trust your keychain you can install things securely: the private pairs are in hands of people that have been through processes similar to this http://www.debian.org/devel/join/newmaint .

What it comes down to is you have to trust someone, eventually. You've got to trust your distro and it's installed verion of GPG, or yo've got to trust your own compiled version of GPG, etc. Unless you're savvy enough to download the source code and confirm it's safe, you're stuck trusting someone.

It's sort of a chicken and the egg problem, PGP. A better web of trust would help resolve these issues (I'm probably not that many trusted steps from a Debian developer), but without a better web of trust, it's hard to build a web of trust, if that makes any sense.

And it seems no one does key signing parties any more...

If they have access to the packages, they have access to the checksums. The checksums only verify that the file isn't corrupt, what you should receive is what you actually received.

True; proper digital signatures would be better. My understanding is the checksums exist to be distributed so that maybe someone would notice "hey, foo.tar.gz on this mirror has the wrong checksum!" The way it works in Homebrew, for instance, is that the sha256 of a file downloaded from the source server is verified against the checksum stored in Homebrew's github repository. Someone would have to compromise both the source server and the github repository to break it. Not ideal, but better than nothing. More importantly it's automatic; Homebrew users don't think about it.

git has pretty good signatures for tags. Maybe there's a way to leverage that for secure open source distribution.

Horde 4 is not affected. If you're running it, you're fine.

The affected releases are:

- Horde 3.3.12 downloaded between November 15 and February 7

- Horde Groupware 1.2.10 downloaded between November 9 and February 7

- Horde Groupware Webmail Edition 1.2.10 downloaded between November 2 and February 7

Well, shit. So much for host-it-yourself services being more secure. Which dogma do I trust now?

Check with: grep -r "\$m\[1\](\$m\[2\])" /path/to/horde

kernel.org, vsftpd, unrealircd, who will fall next?

it's 2012: stop using horde and install roundcube !

Have you tried Horde 4? It is much better than version 3 and is not affected by this issue.

Tried? Yes. Failed? Miserably.

I've happily used Horde for well over 5 years. I was really really looking forward to Horde4. Once it came out, I immediately tried it. Tried to install it, that is. I failed. The lack of clear documentation combined with the dark magic of php-pear kept me from migrating. And even when I did get it installed I was unable to comprehend the UI. It completely changes depending on what 'application' you're using. There is no consistency.

E.g.: The calendar in Horde4. The interface completely changes into nothing you've ever experienced. It's unrecognisable from the rest of Horde4.

I migrated to RoundCube + Plugins for calendar (caldav+davical) and addressbooks (carddav + davical) and have been a happy, android syncing camper ever since.

php-pear dark magic is nothing compared to installing the framework libraries from git! The pear.horde.org instructions fail to mention that you must install horde/Horde_Role before any of the applications.

The UI is currently undergoing a rewrite, but the default interface can be forced into the traditional view in the Horde configuration.

"Horde backdoored" - Hey, that rhymes!