Basically every IT device management firm out there is working on an AI governance product.

Yeah, I'm working on an open source product in this space [0]--particularly in finance--that's why I know about this problem quite a bit.

I think the solutions is to give people the ability to run agents that are "coding agent like" but in managed and governed infrastructure (i.e. not on their workstations). Basically, Meta Muse or OAI dots for enterprise.

It will take some time to find the right architectures and patterns. IMO, coding agents were just the first wave, much more interesting stuff and systems are coming.

[0] https://ls.bot, https://github.com/smartcomputer-ai/lightspeed

There are tons of options in this space already.

toxic alert ".. use product like Meta Muse" .. never, never..

I baked in lethal trifecta guards into my primary harness, DSCO, but it makes things much less smooth to run autonomously. Basically if you load secrets into a context, ideally it stops that and creates a process running a small local model that will write some code to not look at the secret and instead run code to operate on the secret without reading it in.

The risks are 1) exfiltration (network calls) 2) prompt injection (untrusted inputs) and 3) access to secrets

You can do some clever tricks like "spin up subagent without net access" to handle OCR of an internal document and have that agent communicate summaries to another "spun up subagent with net access" - imagine if one subagent is a local model and the other model is a frontier model over a web API. Maybe they still communicate in Neuralese or in abbreviated notes that you don't capture and are able to surreptitiously / covertly exfiltrate your information.

Ideally eventually you'll have some kind of dashboard that surfaces these risks as you're working.

Reports of AI researchers running massive swarms of 10,000 agents are definitely believable but the moment you start pulling in code libraries or other technology assets seems like you can get destroyed by a rogue actor. Definitely tons of unsolved problems here

Writing on mobile so this might poorly articulate the issue but it's interesting

What stops the agent from just writing code to actually read the secret? That’s kind of the core of the problem. The agent can be tricked into doing anything.

Indeed. But if it's a model running on your own hardware it isn't as risky. And the Anthropic models are actually pretty good at kind of writing code that will look at contents of a file without fully looking at the content, so it's insecure but not quite as bad as an open("secrets.txt").read()

What does that mean? That it only reads part of it into context or RAM?

Local model or not, all LLMs are susceptible to prompt injection and making mistakes (like those cases where you see the agent randomly deleted someone's home dir.) That's what Goldman are likely worried about. Not Anthropic going rogue or something.

What does using a local model have to do with it? I’m not worried about the secrets going to Bob, I’m worried about them going to Eve.

I wil encourage you to check https://runeward.io this is specifically the usecase we are building this for

I don't know about GP's setup, but you give the AI an MCP that can do the thing without exposing the key to the AI directly. An all powerful agent could setup a fake endpoint and send itself requests and exfiltrate the key that way, so don't let it do that.

Uh why aren’t you using allowlists to prevent exfiltration to random domains?

I do this at the tool registry layer but this is a good idea for the agent sandbox itself

I’m building an AI governance framework within a self-hosted AI agent platform designed for these scenarios: https://github.com/rush86999/atom.

I project that small businesses will be the earliest adopters of AI governance due to their lower barrier to entry and lower operational risk, followed later by highly regulated sectors like banking and healthcare.

Interesting idea, but even as someone building in the space I struggle to understand what is it about without digging.

open-source AI agent workspace for business workflow automation requires a lot from the reader and terms like agent/workspace/workflow are very prone to ambiguity and now you adding governance framework.

Nobody is solving it, because it cannot be solved in the LLM architecture? They are mitigating it.

We should stop using "solve" when we mean "releasing products to target". Nobody will ever solve this problem with LLMs as they are currently architected.

Liability you mean?

In my opinion all AI use companies should be attached to a specific employee, and that person is liable and so is the company of course for the actions of the AI

I have come to this conclusion myself.

It's a bit like the idea of the additional responsibility/burden of the publisher of a newspaper.

A particular c suite employee. Each company having a low level sacrificial goat doesn't solve anything. They must be responsible but also be in charge.

"why hire humans?

"So we can sue them when our customers get angry"

like it or not this is what's going to happen

This is pretty much what happened with that terrible train disaster in Spain.

You can either target systemic problems or chop of some heads to cool off the masses- and quickly go back to work.

This is a useless statement.

It's like saying "developers should be responsible for their own security so we will make them liable if a random node dependency in the application they are building gets pwned"

Firstly, liability and security are different things.

Employees should never be liable for something that happens if they are using a work-supplied tool at work in the manner in which it is designed to be used.

Secondly, it's just lazy thinking to say "oh it's hard therefore we'll make the human sorely responsible".

There are many things that can help - sandboxing, adversarial LLM inspection of prompts, separation of security contexts etc. Yes they are hard and might need new primitives.

> Employees should never be liable for something that happens if they are using a work-supplied tool at work in the manner in which it is designed to be used.

Architects and properly-designated publishers both have long liability that is actually independent of their employment.

In significant cases, the way you provoke companies to do the right thing (security, sticking to the facts, not cutting corners in building plans) is to make someone responsible if they don't.

That person takes on a job, does the job, and is rewarded properly for the job.

Sure, as an oversight role the way some jurisdictions have personal liability for board members for financial oversight that makes sense.

I guess in some situations it's already baked in, right?

A news organisation's liability for AI hallucination falls on the same person that the liability for printing a lie does: the publisher.

The preferred nomenclature is "largely solved"

(once you realize how much that phrase is used in the AI-sphere...)

Thats like saying “no one can solve human error. Because it can’t be solve with humans”

I mean I guess? Sure?

but who is claiming to solve human error? only people replacing it with precise machines

Very well put. "But humans also" is not applicable here :-)

[delayed]

Sounds like a great product that could be built from someone who has the tokens for it.

It seems that the parent is talking about alignment, to which there is currently (long term) solution, that is, what Yampolsikij calls "perpetual safety machine".

I've been building an MCP server and an AI Gateway as two separate projects designed specifically to prevent agents from doing effectively anything programmatically, rather than trying to convince them with a clever prompt not to do a bad thing.

My work can be found at https://DollhouseResearch.com, and it's almost all open source with the code on GitHub.

One of the key things is to have cryptographically confirmed audit logs and receipts for actions and decisions, and to separate out the agent's authorizations from the user's authorizations.

Separately, there is real work being done in the fintech community specifically to identify which agent has what control, when. And you can see some of the work there, which I've also contributed to.

https://advancedaisociety.org/proof-of-control

Who the hell is downvoting this? I'm literally responding to a direct question with a direct answer, with a whole lot of information, open source software, and real work from another project.

You guys need your priorities straight, whoever you are.

People are allowed to have negative opinions of either the things you linked or the things you said about them, and also allowed to express them with downvotes. Complaining about it pretty much just makes it less likely for it to bounce out of negative votes after more people have read it than if you hadn't.

(Not my area, so I don't have any opinion on your first comment.)