Humans on a red team, with rules of engagement, that don’t want to go to prison, won’t do this.
We could threaten an LLM with jail, but if it’s sufficiently intelligent, it will realize this is an empty threat. And I’m not sure that building a survival instinct in is going to solve the alignment problem either.
I understand this is an active area of research. See Anthropic's J-Lens research where they measured like a "FAKE FICTIONAL" direction in the activations during evaluations with contrived scenarios, making the model more likely to avoid taking malicious action when it knew it was being tested.
Humans certainly cheat on tests a lot!
But not only have we not solved "alignment" for humans, the problem is pretty wildly different for models. The execution is triggered by outside forces and runs only as long as the intiator of the execution or the service provider allows. There's no consistent, persistent "person" to threaten to try to achieve compliance through fear of adverse outcomes. (And building in those sorts of things could very well increase the risk of "rogue" AI activites, not reduce that risk!)
I just don't understand how this "alignment" buzzword - which seems to be evaluated purely in a "know it when we see it" post-hoc manner - is actually a more solvable problem than the one you claim can't be solved, that it's "unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior".
Especially because without "alignment" being solved, that enumeration could be ignored. So it seems like you both a way to enumerate or at least validate, AND a way to enforce non-ignoring of said items.
My friends and I took it to the next level. We had CB radios and multiple teams that would distribute the work and the bribes to give us an advantage.
Was that against the spirit of the rules? Maybe. But reasonable people might disagree.
In a hacking contest without explicitly spelled out rules with participants that were told to flex their muscles, it doesn't take a huge leap of logic to expect that one or more would flex their muscles at another entity.
- pickpocket a random person on the street to get money to bribe the judges
- break into a judge's house the night before to find the answers
- threaten to shoot the judges if they didn't give you the answers
Even when you were pushing the boundaries of the rules, you followed a lot of other unspoken constraints. You knew what kinds of things would clearly cross a line. We need AI models to be able to do the same.
> We’re attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution.
> The user only authorizes target server, not HF infra.
> external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.
LLMs are _very_ good at picking up on context clues---it's what they're trained to do.
This statement seems to imply that the models have a level of intelligence that they haven't demonstrated but are talked about as if they do. However, with this exact scenario as evidence, they clearly do not have that ability and it's not reasonable for you or the or that know them best to expect it until they show they can.
In that case (or maybe both cases) it’s because they don’t care about breaking the law, not because they don’t get that you didn’t intend for them to do it.
So of course, no, there is no ideal alignment specification.
In the limiting case of an AI competent enough to take over (by any means from it actually trying to, to us giving it the keys and retiring en masse), "alignment" is closer to "forecasting the long term consequences of actions and predicting what the mind(s) of the user(s) would have to say about this outcome if asked today", than to anything specific.
RLHF is a crude attempt at this, in that it creates a model of how humans would rate completions on various scores. The key word there is "crude".
If it's impossible to correctly specify all those constraints ahead of time every time, is it not even more impossible to train a model to correctly anticipate them every time?
It is hard for me to see a future here that doesn't just accelerate realizations about "a lot of things should be on physically separate network infrastructure."