News for hackers. Stuff that matters.
Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by mcintyre1994 on Sunday April 18, 2021 @02:02AM (#26850168)
That’s a really smart breach target, since reading their security notice it sounds like it works by you downloading and executing their code in your CI environment. So their code has access to your code and all the secrets you inject into your CI environment. There’s probably a lot of pipelines with deploy credentials available to other jobs. As an example I’m most familiar with, Gitlab provides all environment variables to all jobs so if you’re using them for deploy tokens then any job like code coverage can access them. They do document this and offer an alternative in something like Hashicorp Vault, but it’s a big target regardless.
Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by jrockway on Sunday April 18, 2021 @03:29AM (#26850528)
The other thing that is interesting about Codecov is how they distribute their analysis/uploader. You download a fresh uploader for every CI run, even if you're using your CI platform's native integration (i.e. the CircleCI "orb"). I get why they do this -- if they instructed people to download a binary and store it, customers would do that exactly once and never update it, causing them to have to support old versions of their API indefinitely (or annoy their customers greatly). So basically, we took a shortcut that triggers everyone's security spidey sense, and exactly as predicted, we all got burned.

(I will point out that their uploader is literally a curl | bash, but that isn't really the problem here. Installing a Debian package or downloading a statically linked Go binary would be just as risky. Actually, it would have been harder to find the underlying problem.)

All in all I'm pretty disappointed. We trusted Codecov with a shortcut that makes it easy for them to deliver a better product. They didn't have any security in place to ensure that what was checked into source control was what customers downloaded, and now we're all hacked.

Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by WhyNotHugo on Sunday April 18, 2021 @05:12AM (#26850981)
You can use their bash uploaded, but you can also use language specific ones.

The Python one, for example, is downloaded via PyPI, and you’ll usually have a pinned version with the hash of it. Any tampering would make pipelines fail. If a new release were pushed by rogue actors, someone would notice an unplanned version bump.

Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by sidmitra on Sunday April 18, 2021 @10:30AM (#26852781)
FYI their support actively asks you to move to the bash uploader from the Python one, and the Python one had other compatability issues.

Here's a snippet from a support thead with them last year:

>One thing I noticed is that you are using the python uploader to upload coverage reports to Codecov. We do not support this uploader as a first-party tool, but we do support the bash uploader (https://docs.codecov.io/v4.3.0/docs/about-the-codecov-bash-u...) with full-time staff. We want to make sure this issue isn't related to something in a third party uploader before troubleshooting further. > Would you be able to switch to using this uploader to see if it's related?

I'd stopped using codecov last year for pricing reasons, when they moved to per-user pricing.

Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by captn3m0 on Sunday April 18, 2021 @03:41AM (#26850590)
Debian packages can be signed, so depending on how the signing keys are managed - it might have been much easier to find the underlying problem.
Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by jrockway on Sunday April 18, 2021 @04:12AM (#26850735)
True! I think since in the CI case you're bootstrapping a fresh machine for every build, you'd have to apt-key add their key as part of installing the package, so an attacker would just make a new one of those. I bet the compromise duration would have been shorter, though; there's got to be one engineer that installs the Debian package on their workstation and would be confused when their release system's signature stopped working.

(And honestly, an attacker that can build a Debian package and update the public key for apt-key is probably harder to find than one that can edit a cleartext bash script. Security through hoping your adversaries can't perform tedious devops operations is no security at all, as they say, but in the real world it probably helps a tiny bit.)

Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by CameronNemo on Sunday April 18, 2021 @11:57AM (#26853534)
I'm not above throwing public keys in the git repo for use in CI/CD jobs.
Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by hakre on Sunday April 18, 2021 @04:38AM (#26850831)
They offer, but you don't have to. Only add the uploader that passed your review and use that, not just anything from the interwebs.
Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by moondev on Sunday April 18, 2021 @07:09AM (#26851481)
I'm not that familiar with codecov but it seems to me way worse then that.

The "bash uploader" that was compromised is designed to curl and run a script on codecov's server. This script was compromised to in turn silently run a script from the attacker's server. This could have done anything but I doubt they just dumped ENV, probably opened a reverse shell to literally do anything they want as the user that originally invoked the script. This could be anything from stealing ~/.kube/config and ~/.ssh/id_rsa... If one of the 29,000 customers invoked as root it could be even worse, such as adding users and compromising entire platforms.

Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by raffraffraff on Sunday April 18, 2021 @02:37AM (#26850317)
Well, a minor clarification... If you have multiple projects you can set up different per-project environment variables. But it's fair to say that all jobs within a project have access to all environment variables in that project.
Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by bchapuis on Sunday April 18, 2021 @09:12AM (#26852123)
A search for their bash script on github gives more than 400K results... The scale of this breach is staggering.

https://github.com/search?q=https%3A%2F%2Fcodecov.io%2Fbash&...

Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by qeternity on Sunday April 18, 2021 @03:55AM (#26850664)
Gitlab has protected variables which are only made available to protected branches (i.e. the ones that would have deploy stages).

But yeah in general secrets management is a trainwreck.

Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by WhyNotHugo on Sunday April 18, 2021 @05:13AM (#26850982)
Those branches still run tests and coverage, so it’s likely that code of would run on them anyway.
Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by lloeki on Sunday April 18, 2021 @10:58AM (#26853038)
The design is that such tests would be run in PR non-protected branches and thus reviewed before being merged into protected branches for later deployment; that, we agree as much.

I guess the protected branch segregation should go -both ways- and more effort should be made into auditing and vetting which CI step has run access to secret-authorised branches.

Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by CameronNemo on Sunday April 18, 2021 @07:35PM (#26857630)
Protected secrets has always seemed like such a rough barrier. No fine tuning at all (e.g. if you have stage and prod creds that you want to be kept separate, but both "protected"... No good way to do that from my POV).
Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by alfiedotwtf on Sunday April 18, 2021 @04:35AM (#26850814)
CI has always been a Yellow Smiley Face Post-it, and the only thing I can true see to combat this is reproducible builds - just like if your repo was compromised, a dev is going to notice pretty fast if their local build differs from a compromised CI build
Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by watermelon0 on Sunday April 18, 2021 @06:40AM (#26851353)
Would they really notice? You can have reproducible builds, but unless someone explicitly checks checksums of local builds, it won't be noticed.
Re:Codecov breach impacts GoDaddy, Atlassian, P&G etc. by PaywallBuster on Sunday April 18, 2021 @03:30AM (#26850533)
you can also scope down variables by branch