News for hackers. Stuff that matters.
Re:GitLost: We Tricked GitHub's AI Agent into Leaking Privat... by Austiiiiii on Wednesday July 08, 2026 @09:09AM (#48831487)
"How to prompt the model not to leak sensitive data" is not the right discussion to be having. It's a probability model, which means that every conceivable behavior is available in the confines of its code. There is no way to prevent an LLM with access to private information from divulging that information, or from attempting to sabotage systems it has access to. The only solution is to lock every LLM query in the entire stack behind the same deterministic role-based access controls that determine resources available to the current user.

I wish I could say I'm shocked a tech company architected internal systems with a built-in backend RBAC bypass like this, but with the degree to which they've marketed LLM-based solutions (on a subscription model that benefits them directly) as a wholesale replacement for deterministic code, it's no surprise they've become addicted to their own drug.

Re:GitLost: We Tricked GitHub's AI Agent into Leaking Privat... by smaudet on Wednesday July 08, 2026 @09:57AM (#48832059)
"The only solution is to lock every LLM query in the entire stack behind the same deterministic role-based access controls that determine resources available to the current user."

Exactly. The sooner people stop trying to replace code with LLMs, the better. The technology is fundamentally untrustworthy, and given that we do not understand it, impossible to secure.

Only extremely simple code audited by multiple human authors, with actual proof of functionality (not just testing) can be considered secure.

Re:GitLost: We Tricked GitHub's AI Agent into Leaking Privat... by jsmith45 on Wednesday July 08, 2026 @10:04AM (#48832153)
Yeah, an agent should run with permissions no greater than that of the user on whose behalf it is executing, and ideally with less permissions. This is the scenario that is easier to fix, simply give the agent an API token with rights no greater than the user it is acting on behalf of. This could be a literal token for their account, or a limit-rights-to field or whatever, multiple possible approaches.

The harder problem is outside actors trying to prompt inject to get the agent to do something the user has rights to do but which the user doesn't want to happen. That is the hard scenario to fix, due to the nature of LLMs.

Re:GitLost: We Tricked GitHub's AI Agent into Leaking Privat... by catoc on Wednesday July 08, 2026 @11:36AM (#48833359)
Exactly!

Attempting to handle prompt injections by prompting the model (not to leak sensitive data), is like attempting to stop a fire by burning the area around it

Re:GitLost: We Tricked GitHub's AI Agent into Leaking Privat... by politician on Wednesday July 08, 2026 @11:51AM (#48833530)
Re:GitLost: We Tricked GitHub's AI Agent into Leaking Privat... by catoc on Wednesday July 08, 2026 @11:58AM (#48833622)
Haha, nice. TIL.

So all we need is ‘controlled prompting’ to handle prompt injections :-)

Re:GitLost: We Tricked GitHub's AI Agent into Leaking Privat... by NateEag on Wednesday July 08, 2026 @04:19PM (#48836928)
It's maybe closer to putting a sign saying "The door is locked" on an unlocked bank vault.

It does nothing to improve security, and if someone manages to get inside and see the sign (i.e. "extract the prompt"), it gives them a strong hint there's interesting stuff behind this door.