It should be reserved for a human because humans have time preference. An AI will wait for your prompt for a billion years. It can get turned off and found by alien archeologists a million years in the future, turned on and it won't notice. Biological entities care about time a lot though, and it is at the core of most human ethics and values and ultimately real and can't be denied. Even the most ardent solipsist or moral relativist fears spending years in prison.

I'm struggling to imagine aliens resurrecting a data center and countless hackneyed python packages so they can spew a language they don't even know. Not that it wouldn't be useful to their linguists, of course, but still a very optimistic analogy for LLMs.

It should be reserved for a human because humans have a body and a body can be put in prison.

A bit dramatic for effect but true.

Yes, this.

For a slightly less dramatic version - you can fire a human if they consistently do the wrong thing despite being told how to do it better.

Putting a big ball of matrix arithmetic on a Performance Improvement Plan makes no sense.

I agree with this. I basically don’t feel comfortable trying to pretend a fancy matrix multiplication can be responsible for anything.

But what if companies that don’t track responsibility outcompete those who do?

In particular what if some perfect AI decision making ends up nailing decisions that maximize the expected reward for the company. And, well, if that comes at the cost of some unmanaged low-probability catastrophic risk, the company doesn’t care because all the decision makers are AI that don’t mind being shut off.

I don’t know, I can imagine that the equivalent of a PIP for an agent would be noticing flaws in its output and then creating additional evals, modifying the harness, upgrading the model, etc. to improve its performance. That’s not SO different from the PIP, except that the LLM doesn’t really care in the same way?

The person making the tweaks to the harness is the one taking accountability for fixing those mistakes - they're the DRI in this scenario.

Like the manager being responsible for someone on a PIP.

Obviously they’re different, but it’s interesting to think about what is unique about humans that makes them able to be accountable or responsible in a way that LLMs cannot be. Is it at core just that they can be fired? Or essentially the threat of suffering?

I think just the possibility of consequences that they can give a damn about. An LLM doesn't have feelings no matter how much human-like text it can simulate. There is literally nothing going on between prompts for any given model. They are incapable of worry or any other emotion. Wipe the context clean and the LLM is completely unaware there was ever a problem.

you can make an AI operate on time scale...

unfortunately,

what's developing is more about scape goats than anything rational like responsibility.

It’s a double edged sword. You are responsible for a thing. But you can’t just wave a wand and say “x team needs to do y for the thing I am a DRI for”. There are some resources in Apple University (an internal learning resource) about navigating it. “Influence without control” and while it’s helpful, they acknowledge it sucks sometimes.

Excellent! We can use this to start to throw executives in jail and maybe society won't erode as fast.

A security measure I use on my projects is never to give the AI models in production write authority at all - AI simply produces the "suggestion" in a structured format and then deterministic code validates it by making sure it conforms to pre-set constraints. This way, the blast radius of a really dumb decision is minimized. I do the same thing essentially in the codebase as well where we have architectural static analyzers that box in the agents and limit how stupid they can be with their changes.