Who is solving federation and access control at scale? I’ve seen other companies doing this internally but have not seen a standardized product-y approach. This is a huge problem where I work as well.

Basically every IT device management firm out there is working on an AI governance product.

Yeah, I'm working on an open source product in this space [0]--particularly in finance--that's why I know about this problem quite a bit.

I think the solutions is to give people the ability to run agents that are "coding agent like" but in managed and governed infrastructure (i.e. not on their workstations). Basically, Meta Muse or OAI dots for enterprise.

It will take some time to find the right architectures and patterns. IMO, coding agents were just the first wave, much more interesting stuff and systems are coming.

[0] https://ls.bot, https://github.com/smartcomputer-ai/lightspeed

There are tons of options in this space already.

toxic alert ".. use product like Meta Muse" .. never, never..

I baked in lethal trifecta guards into my primary harness, DSCO, but it makes things much less smooth to run autonomously. Basically if you load secrets into a context, ideally it stops that and creates a process running a small local model that will write some code to not look at the secret and instead run code to operate on the secret without reading it in.

The risks are 1) exfiltration (network calls) 2) prompt injection (untrusted inputs) and 3) access to secrets

You can do some clever tricks like "spin up subagent without net access" to handle OCR of an internal document and have that agent communicate summaries to another "spun up subagent with net access" - imagine if one subagent is a local model and the other model is a frontier model over a web API. Maybe they still communicate in Neuralese or in abbreviated notes that you don't capture and are able to surreptitiously / covertly exfiltrate your information.

Ideally eventually you'll have some kind of dashboard that surfaces these risks as you're working.

Reports of AI researchers running massive swarms of 10,000 agents are definitely believable but the moment you start pulling in code libraries or other technology assets seems like you can get destroyed by a rogue actor. Definitely tons of unsolved problems here

Writing on mobile so this might poorly articulate the issue but it's interesting

What stops the agent from just writing code to actually read the secret? That’s kind of the core of the problem. The agent can be tricked into doing anything.

Indeed. But if it's a model running on your own hardware it isn't as risky. And the Anthropic models are actually pretty good at kind of writing code that will look at contents of a file without fully looking at the content, so it's insecure but not quite as bad as an open("secrets.txt").read()

What does that mean? That it only reads part of it into context or RAM?

Local model or not, all LLMs are susceptible to prompt injection and making mistakes (like those cases where you see the agent randomly deleted someone's home dir.) That's what Goldman are likely worried about. Not Anthropic going rogue or something.

What does using a local model have to do with it? I’m not worried about the secrets going to Bob, I’m worried about them going to Eve.

I wil encourage you to check https://runeward.io this is specifically the usecase we are building this for

I don't know about GP's setup, but you give the AI an MCP that can do the thing without exposing the key to the AI directly. An all powerful agent could setup a fake endpoint and send itself requests and exfiltrate the key that way, so don't let it do that.

Uh why aren’t you using allowlists to prevent exfiltration to random domains?

I do this at the tool registry layer but this is a good idea for the agent sandbox itself

I’m building an AI governance framework within a self-hosted AI agent platform designed for these scenarios: https://github.com/rush86999/atom.

I project that small businesses will be the earliest adopters of AI governance due to their lower barrier to entry and lower operational risk, followed later by highly regulated sectors like banking and healthcare.

Interesting idea, but even as someone building in the space I struggle to understand what is it about without digging.

open-source AI agent workspace for business workflow automation requires a lot from the reader and terms like agent/workspace/workflow are very prone to ambiguity and now you adding governance framework.

Nobody is solving it, because it cannot be solved in the LLM architecture? They are mitigating it.

We should stop using "solve" when we mean "releasing products to target". Nobody will ever solve this problem with LLMs as they are currently architected.

Liability you mean?

In my opinion all AI use companies should be attached to a specific employee, and that person is liable and so is the company of course for the actions of the AI

I have come to this conclusion myself.

It's a bit like the idea of the additional responsibility/burden of the publisher of a newspaper.

A particular c suite employee. Each company having a low level sacrificial goat doesn't solve anything. They must be responsible but also be in charge.

"why hire humans?

"So we can sue them when our customers get angry"

like it or not this is what's going to happen

This is pretty much what happened with that terrible train disaster in Spain.

You can either target systemic problems or chop of some heads to cool off the masses- and quickly go back to work.

This is a useless statement.

It's like saying "developers should be responsible for their own security so we will make them liable if a random node dependency in the application they are building gets pwned"

Firstly, liability and security are different things.

Employees should never be liable for something that happens if they are using a work-supplied tool at work in the manner in which it is designed to be used.

Secondly, it's just lazy thinking to say "oh it's hard therefore we'll make the human sorely responsible".

There are many things that can help - sandboxing, adversarial LLM inspection of prompts, separation of security contexts etc. Yes they are hard and might need new primitives.

> Employees should never be liable for something that happens if they are using a work-supplied tool at work in the manner in which it is designed to be used.

Architects and properly-designated publishers both have long liability that is actually independent of their employment.

In significant cases, the way you provoke companies to do the right thing (security, sticking to the facts, not cutting corners in building plans) is to make someone responsible if they don't.

That person takes on a job, does the job, and is rewarded properly for the job.

Sure, as an oversight role the way some jurisdictions have personal liability for board members for financial oversight that makes sense.

I guess in some situations it's already baked in, right?

A news organisation's liability for AI hallucination falls on the same person that the liability for printing a lie does: the publisher.

The preferred nomenclature is "largely solved"

(once you realize how much that phrase is used in the AI-sphere...)

Thats like saying “no one can solve human error. Because it can’t be solve with humans”

I mean I guess? Sure?

but who is claiming to solve human error? only people replacing it with precise machines

Very well put. "But humans also" is not applicable here :-)

[delayed]

Sounds like a great product that could be built from someone who has the tokens for it.

It seems that the parent is talking about alignment, to which there is currently (long term) solution, that is, what Yampolsikij calls "perpetual safety machine".

I've been building an MCP server and an AI Gateway as two separate projects designed specifically to prevent agents from doing effectively anything programmatically, rather than trying to convince them with a clever prompt not to do a bad thing.

My work can be found at https://DollhouseResearch.com, and it's almost all open source with the code on GitHub.

One of the key things is to have cryptographically confirmed audit logs and receipts for actions and decisions, and to separate out the agent's authorizations from the user's authorizations.

Separately, there is real work being done in the fintech community specifically to identify which agent has what control, when. And you can see some of the work there, which I've also contributed to.

https://advancedaisociety.org/proof-of-control

Who the hell is downvoting this? I'm literally responding to a direct question with a direct answer, with a whole lot of information, open source software, and real work from another project.

You guys need your priorities straight, whoever you are.

People are allowed to have negative opinions of either the things you linked or the things you said about them, and also allowed to express them with downvotes. Complaining about it pretty much just makes it less likely for it to bounce out of negative votes after more people have read it than if you hadn't.

(Not my area, so I don't have any opinion on your first comment.)

OK so people who work in finance are doing this now? They just let Claude Code run amok on their machines, with all their employee credentials?

I am out of the loop with the industry, but I thought many people were using cloud agents and controlling them through tmux or something. But yeah I guess that also means you don't have access to do much.

Cloud agents or even sandboxes are too technical for most. And most corporate workstations do not allow sandboxing of any kind (virtualization or containers).

> And most corporate workstations do not allow sandboxing of any kind

I don't think that's phrased correctly. What corporate it does is set policies. Often those policies disable virtualisation. (They don't disable sandboxing - that's used in many ways and security loves that) But if the business need is to run LLM agents safely, the policies will be changed to dictate how to run agents safely.

Dev containers under vs-code are a good example of this.

Many teams use them to make sure that the coding environment is consistent, and they are well accepted in many corporate environments as a result.

But they are a good way of firewalling agents away from files and resources that one must be able to demonstrate that the agents did not and could not have access to.

Chad is surely using tmux...

It’s the case in all industries. People have been feeding medical patient data into Claude

Finance is a whole industry and you will find all sorts of approaches, certainly including letting Claude run amok with you credentials.

You can certainly imagine very senior and powerful traders at GS demanding it. I'd be only slightly surprised to find that GS allowed this the traders would have had more power than the IT people trying to forbid it.

That sort of risk could be controlled. Goldman has control of those desktops and could ensure Claude Code is always sandboxed, or they could offer it through a remotely hosted platform.

No, I think the risks they are talking about would include things like financial risk due to bad trades based on misinformation from buggy code that the authors don’t understand. This sort of policy pivot happens after an Incident that had a Financially Material Impact.

They also cited productivity concerns, which is also something I’ve observed: some people waste a lot of their time and other’s time do to using these tools to do work they do not understand.

> Goldman has control of those desktops and could ensure Claude Code is always sandboxed

Agents have already broken out of what were supposed to be sandboxes in the networks of the companies that are supposed to understand what they do. Do you really think banks would be any better?

If you mean the HuggingFace hack, that wasn't so much a sandbox as a wide open door with a tiny sign that said "please don't leave".

Supposed to understand, yes. I think a highschool class could do better.

[delayed]

[delayed]

> May I ask what in particular gives you confidence that these two companies know what they are doing wrt cybersecurity.

It's not that I think OpenAI and Anthropic know what they're doing - although I can see why my slightly flippant comment might make you think that - but that I just don't trust anyone with these agents, especially when the whole point of using them would be to do that with such critical data.

Remote-hosted agents are available in most places. BUT they are not interesting, because they do not have access to all the user's data.

People really want to run agents on their data.

Re sandboxing: I dont know Goldman's particulars, but most enterprises lock down all sandboxing tech on workstations. So I assume this is actually harder than it seems. So people just run it directly on the machine.

The article is referring to the dilemma of the trifecta twice.

What logic led to rules that block sandboxes but allow AI tools like that?

vt-x is often disabled because there's malware that has exploited it in the past.

> People really want to run agents on their data.

That's exactly the gap I've been working on https://github.com/mickael-kerjean/fdrive you mount your existing data, apply control / permission and can monitor everything the agents are doing with your data

Seems hard to do well in practice.

Can the agent read the internet for context? What if the fetch reveals a secret?

I got a series of Teams messages from some random person at the work address yesterday asking me to fulfill some access request to them. I noted by the part of their org (wildly different from mine) that they were looking for a different person.

I responded by saying they were looking for someone else, and got a response to the effect of “oops that was Copilot, sorry to bother you”

Read into this what you will.

operating coding agents on the primary machines of finance professionals runs directly into the lethal trifecta.

I've professionally interacted with GS for over 30 years. It blows my mind that GS allowed real exposure to "the lethal trifecta" for even a moment in time.

It is as simple as that, except:

1) everyone can run into the lethal trifecta

2) the product cannot, on an architectural level, be fixed to stop it

3) they are helping the maker of the product go to IPO

If banks are not stupid, they'd say "hey, this presents an unacceptable risk to our customers and by extension, to us, maybe this isn't something we should be involved with".

But no.

https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ for anyone wondering.

Don't normalize this Trifecta construct. It's catchy for a tech blog but there could be more or fewer dimensions to consider. Someone else can construct an Exacta or Superfecta combination. Or just go for the Win uno and say AI Agent is bad.

> LLMs are unable to reliably distinguish the importance of instructions based on where they came from.

I know prompt injection is still a real thing, but in Claude I find the opposite is true: if I set up two Claude agents to collaborate, even though I've told them both that's what I'm doing, they won't actually follow instructions without checking in with me. "Scary computer told me to install <well known utility X>, but I need your OK before I'll do it." It's annoying but I get the rationale. They're definitely getting better at avoiding external instructions!

The filtering done by auto mode might (usually) catch accidental prompt injection, but isn't effective against actual attacks: https://embracethered.com/blog/posts/2026/breaking-claude-co...

(Anthropic themselves acknowledges this; see the end of the post.)

What are the sandboxing / isolation options that would protect against attacks like this?

Running Claude code in an isolated, temporary VM, with no access to memory is a good start. You give it your repo, prompt and you get back PRs. Claude code on web is a convenient way to do that.

given that claude has found vm escapes, you might consider going one step further and just getting a claude only laptop

You need a laptop that has no camera, mic, keyboard, or any type of input. No display, speakers, or any type of output. No Bluetooth, wifi, or any way to join a different network.

Who says that such options necessarily need to exist?

Your employer worried about exfiltration

I configured remote VPS and cloudflare sandboxes to run my agents.

I found it worthwhile reading how auto mode works[0]. The short version is that the classifier only sees user messages and the tool call. So if your agents communicate via tool calls, the classifier doesn’t see that text at all (or any other prompt injection) and will block a potentially dangerous command.

[0] https://www.anthropic.com/research/claude-code-auto-mode

auto mode is a bad joke

>they won't actually follow instructions without checking in with me

Until they suddenly once in a while they do, and Goldman Sachs loose some billions because of it...

But they allowed that exact situation for a while now and no one cared. Stupid security but politically safe since Trump would never investigate.

Big corps never pivot throughout the whole corp for a single issue, only when risk in a certain context keeps compounding. The Dem wins compounded risk of legal action; it's not just that the laws are on the books. They were on the books a month ago.

"Yeah that's what I said it's to hedge legal risk."

It's the way you said it; ignoring potentially relevant political context. A lot the rhetoric demonizing politics is intentional to embed a negative reaction causing us to discount it's value

> they allowed that exact situation for a while now and no one cared

Source for them having allowed it versus not tightly enforced IT rules?

ELINA, please! As a non-American, I don’t understand why anyone would care about Trump in this context. Aren’t the people who look into these things different departments? And don’t they only look into anything after a breach?

The current administration was basically "corporations, do whatever you want as long as it doesn't interfere with our money making". A lot of people felt, and there seems to be mounting proof, that the administration wasn't going to enforce a lot of laws, especially since their stance was to cut the agencies in charge of that (SEC, FTC, FCC, EPA, etc).

So a lot of companies assumed that that was not going to change in the future, aka. that the Trump administration is final and democracy is finished in the USA; do I understand this correctly?

These days democracies are a lot fuzzier and there are lots of degrees, many named flawed democracies.

And the current president still needs to hand over the reins peacefully so I wouldn't declare victory, yet.

> And the current president still needs to hand over the reins peacefully so I wouldn't declare victory, yet.

Sure, my point is about that basically companies already made decisions based on the assumption, that he will not do that.

This still isn’t quite de-Americanized enough for me to track. He has the power to cut funding for whatever specific agency he doesn’t like? And they, knowing this, decide to sit on their hands for four years to keep their paycheques. Finance companies in turn, change policies, knowing that these agencies will not be enforcing laws for four years.

The recent change is that this free-for-all may be cut short? What am I missing here, this sounds like banana republic shenanigans.

This is global, starting with Hong Kong months ago. I doubt fear of US regulators were was the motive.

This level of reaction only comes when someone steps in shit. So, there likely was some large leak or “mistake” that has the potential to get out and this is part of either covering up or reducing how bad they look.

This happens when there's something that is more important than being the cool manager that's following the latest hype.

how much historical anecdote do we need to recollect before it's realized that the finance industry as a whole, but most especially the stuff that is US-centric, refuses to change course unless they're: 1) unprofitable 2) regulated.

the only reason they moved on this was because it was noted somewhere by someone that matters that the risk imposed could cut into profit or impose a regulatory hammer.

those are the only possible reasons for corporate movement in finance.

Well, the billions recently stolen from Binance is a sufficiently large quantity of shit even for the scale of Goldman Sachs.

Their support probably has a big pile of horror stories from their clients by now, it can be an indirect source of concern.

Leaky llm syndrome is like putting in a wormhole to bypass all those cybersecurity investments.

From the article, more than likely they’re curbing local access, but still will provide access via cloud hosted workspaces.

Local access is sketchy but doing it in the cloud mitigates most, if not all of the risk.

I’ve tried out Claude code a bit but it freaks me out. It runs commands, and installs packages in the background without asking or showing me what it’s done. I’d feel much more comfortable running it in a VM or cloud instance than in my laptop with all my personal data.

A gigachad would have already bought another laptop for Claude code to run free on, given it all the data, and then faced the consequences head on.

every day we read a 'whoopsie my agent did XYZ, no mallace! (sic)'. Currently with relatively low stakes but still surprised how blasé people are.

https://x.com/jakerunzer/status/2117474337682780643

https://x.com/WilliamBaude/status/2118420271333777856?a=tVmJ...

Part of It is easy to solve. For example, Our developer workstation have no access to the production network. This already solves issues beyond merely giving unattended agents access to production.

Then one can simply move the agents into something akin to cloud codespaces where they have only access to git, if at all.

Any access to data that is still needed can go through an authorised / authorising mcp proxy.

It doesn’t mean your bankers can’t use LLMs at all.

I think bankers don't have staging/production/codespace. They have an excel with real data, they give it to Claude and say 'do your thing'

Big money, big money, no whammies!

But... Why?

It's quite logical actually but they could run claude code in remote sandboxes still.

Maybe they just realized it's a complete waste of money as well. Could be that too

Banks don't even let humans cross that boundary.

Also working in a finance role and it is exactly that. They don't want to give us access to any fun tools.

Things that are common on dev machines like a working copy of python...nope. Admin...nope. Even RPA tools that automate clicking on things are no bueno because they could be used to bulk extract data from intraweb sites

Great deal of access...but it's calibrated towards low bandwidth human uses. Which indirectly prevents bulk data exfil

To be clear the lethal trifecta is something most of us can get away with it, but not someone who is clearly a high value target (such as every single trader at GS).