It was the PYPI_PUBLISH token which was in our github project as an env var, that got sent to trivvy.

We have deleted all our pypi publishing tokens.

Our accounts had 2fa, so it's a bad token here.

We're reviewing our accounts, to see how we can make it more secure (trusted publishing via jwt tokens, move to a different pypi account, etc.).

How did PYPI_PUBLISH lead to a full GH account takeover?

Don't hold your breath for an answer.

I'd imagine the attacker published a new compromised version of their package, which the author eventually downloaded, which pwned everything else.

Their Personal Access Token must’ve been pwned too, not sure through what mechanism though

They have written about it on github to my question:

Trivvy hacked (https://www.aquasec.com/blog/trivy-supply-chain-attack-what-...) -> all circleci credentials leaked -> included pypi publish token + github pat -> | WE DISCOVER ISSUE | -> pypi token deleted, github pat deleted + account removed from org access, trivvy pinned to last known safe version (v0.69.3)

What we're doing now:

    Block all releases, until we have completed our scans
    Working with Google's mandiant.security team to understand scope of impact
    Reviewing / rotating any leaked credentials

https://github.com/BerriAI/litellm/issues/24518#issuecomment...

69.3 isnt safe. The safe thing to do is remove all trivy access. or failing that version. 0.35 is the last and AFAIK only safe version.

https://socket.dev/blog/trivy-under-attack-again-github-acti...

I have sent your message to the developer on github and they have changed the version to 0.35.0 ,so thanks.

https://github.com/BerriAI/litellm/issues/24518#issuecomment...

Does that explain how circleci was publishing commits and closing issues?

Perhaps it's too obvious but ... just running the publish process locally, instead of from CI, would help. Especially if you publish from a dedicated user on a Mac where the system keychain is pretty secure.

I'm not sure how. Their local system seems just as likely to get compromised through a `pip install` or whatever else.

In CI they could easily have moved `trivy` to its own dedicated worker that had no access to the PYPI secret, which should be isolated to the publish command and only the publish command.

User isolation works, the keychain isolation works. On macOS tokens stored in the keychain can be made readable only by specific apps, not anything else. It does require a bit of infrastructure - ideally a Mac app that does the release - but nothing you can't vibe code quickly.

That's true, but it seems far more complex than just moving trivy to a separate workerflow with no permissions and likely physical isolation between it and a credential. I'm pretty wary of the idea that malware couldn't just privesc - it's pretty trivial to obtain root on a user's laptop. Running as a separate, unprivileged user helps a ton, but again, I'm skeptical of this vs just using a github workflow.

I'm looking for more general solutions. "Properly configure Trivy" is too specific, it's obvious in hindsight but not before.

Privilege escalation on macOS is very hard indeed. Apple have been improving security for a long time, it is far, far ahead of Linux or Windows in this regard. The default experience in Xcode is that a release-mode app you make will be sandboxed, undebuggable, have protected keychain entries other apps can't read, have a protected file space other apps can't read, and its own code will also be read-only to other apps. So apps can't interfere with each other or escalate to each other's privileges even when running as the same UNIX user. And that's the default, you don't have to do anything to get that level of protection.

Privesc is trivial on every desktop OS if you run as a regular user. I can write to your rc files so it's game over.

App Store apps are the exception, which is great, but presumably we're not talking about that? If we are, then yeah, app stores solve these problems by making things actually sandboxed.

Any app can be sandboxed on macOS and by default newly created apps are; that's why I say if you create a new app in Xcode then anything run by that app is sandboxed out of the box. App Store enforces it but beyond that isn't involved.

I feel like we're just talking about different things? I've just said that I'm aware of apps being sandboxed, that does not mean that some random program you run from your terminal is sandboxed.

Right, I'm skipping a step.

What I'm saying is that it's very easy now to take some arbitrary task - doing a compile/release cycle for example - and quickly knock up a simple signed macOS .app that sandboxes itself and then invokes the release script as a subprocess. Sandboxing is transitive and the .app itself can authenticate to the OS to obtain creds before passing them to the subprocess.

In the past I've thought about making a quick SaaS that does this for people so they don't have to fiddle with it locally and maybe some day I still will. But you can easily do it locally especially with Xcode and AI now. You wouldn't have to know anything about macOS development.

Ah, yes. I totally agree with that and wish that's how people built software.

This problem is solved by not having a token. Github and PyPI both support OIDC based workflows. Grant only the publish job access to OIDC endpoint, then the Trivy job has nothing it can steal.

Are you spelling it with two vs on purpose?

>I am unable to understand how it compromised your account itself from the exploit at trivvy being used in CI/CD as well.

Token in CI could've been way too broad.

He would have to state he didn't in fact make all those commits and close the issue.