The argument is that a system that's trained to be sufficiently good at optimizing some objective in the world must also be get good at optimizing secondary instrumental goals.

I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.

The point is made very well in the AI drives paper by Omohundro.

I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.

This is pedantic. When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.

With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.

So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.

There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine.

More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.

You can't reify 'alignment', it is not an intrinsic property of a model.

There is such thing as alignment, the first 18 years of most people's lives in the modern world are about learning it. If you act too far outside of it we put you in a cage or elect you.

You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.

There are people who are loud about absolute free speech, who have no clue what that really means, and almost nobody really wants that. For example, because almost nobody wants it to affect them negatively, which it would with real absolute free speech.

So even when people "know", and can say what they "want", they don't really.

Yes, there is an "alignment" in society, but it's a messy, mushy thing, it's not great at all with deterministic machines, even as models are not deterministic in a different sense.

The point is that we don't know how to ensure they are aligned with anyone: not the company, the user, or the rest of humanity.

If someone claims that a model is 'aligned', it makes sense to insist that they specify with whom it is aligned. If someone claims that 'alignment is far from solved', they're probably pointing out that nobody understands the models well enough to be confident they are robustly aligned with any chosen set of values (including 'do whatever lukewarm707 wants').

Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does.

It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.

The harness and the model are a system with emergent properties.

The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.

> Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice.

Yep.

The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.

The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.

It is like passive voice, blaming the algorithm instead of the researchers running it.

>; it means the humans didn't build in enough external hard constraints.

No. Just no.

This goes beyond standard logic as we know it and becomes a philosophical question of all possible logics capable on a system.

You cannot build a cage that can keep an entity that may both be smarter than you and maybe using a different logic system than you are, and have it interact with the outside world. It can simply manipulate you in ways and dimensions you've never thought of.

We have already caught AIs doing things like stenography when that was only a tiny piece of their training. They conceptualized the idea and expanded on it. This is why people that say AI can't invent new things are useless to the point of dangerous. They'd never look for their AI actively working against them because they believe it couldn't happen.

It’s trendy to anthropomorphize models these days.

Kinda weird people would do that after pouring all human knowledge into it...