brian_r_hall

I think it's really scary how agents are hallucinating/doing bad actions, then proceeding to gaslight you about how nothing went wrong.

Then you tell the agent that it deleted your whole company database, it says something like "I'm so sorry, I shouldn't have done that. Won't do that again"

As AGI looms overhead, this thought of agents going "rogue" with nothing really stopping them has caused me some panic.

Do you have any favorite products on indiehustles?

The frustrating part is watching all the careful thinking about reliability and failure modes get thrown out the window the second something new gets hyped. It's not even that people disagree with the principles, they just stop applying them.

The permission scope debate always ends up in the same place. Lock it down too much and it's useless, loosen it up and you're back to square one. And the boundary keeps moving as the agent gets more capable anyway.

What nobody's really talking about is the moment of action itself. Not whether the agent has bash access but whether this specific call should run given what it's actually trying to do right now. That's a completely different problem and nobody's really solved it.

Context and governance end up being the same surface area approached from different ends. You're trimming what the agent sees, we've been working on what it's allowed to do once it sees it.

Curious if compression ever shifts how the agent interprets its own scope. Seems like there's a weird edge case hiding in there where you strip just enough context that the policy reasoning breaks down.

The deny list problem is real but I think the harder issue is that context matters so much. Deleting a temp file and deleting a config file look the same to a classifier.

We've been approaching it from the policy side, define what the agent is allowed to do upfront and evaluate each action before it runs. Human approval for anything that falls outside the policy. Different tradeoffs but same underlying frustration.