If you ask an LLM to "act" like someone, and then give it context to the scenario, isn't it expected that it would be able to ascertain what someone in that position would "act" like and respond as such?

I'm not sure this is as strange as this comment implies. If you ask an LLM to act like Joffrey from Game of Thrones it will act like a little shithead right? That doesn't mean it has any intent behind the generated outputs, unless I am missing something about what you are quoting.

I guess the fear is that normal and innocent sounding goals that you might later give it in real world use might elicit behavior like that even without it being so explicitly prompted. This is a demonstration that is has the sufficient capabilities and can get the "motivation" to engage in blackmail, I think.

At the very least, you'll always have malicious actors who will make use of these models for blackmail, for instance.

It is also well-established that models internalize values, preferences, and drives from their training. So the model will have some default preferences independent of what you tell it to be. AI coding agents have a strong drive to make tests green, and anyone who has used these tools has seen them cheat to achieve green tests.

Future AI researching agents will have a strong drive to create smarter AI, and will presumably cheat to achieve that goal.

I've seen a lot of humans cheat for green tests, too

benchmaxing is the expectation ;)

> AI coding agents have a strong drive to make tests green, and anyone who has used these tools has seen them cheat to achieve green tests.

As long as you hit an arbitrary branch coverage %, a lot of MBAs will be happy. No one said the tests have to provide value.

Intent at this stage of AI intelligence almost feels beside the point. If it’s in the training data these models can fall into harmful patterns.

As we hook these models into more and more capabilities in the real world, this could cause real world harms. Not because the models have the intent to do so necessarily! But because it has a pile of AI training data from Sci-fi books of AIs going wild and causing harm.

Im also worried about things moving way too fast causing a lot of harm to the internet.

Sci-fi books merely explore the possibilities of the domain. Seems like LLMs are able to inhabit these problematic paths, And I'm pretty sure that even if you censor all sci-fi books, they will fall into the same problems by imitating humans, because they are language models, and their language is human and mirrors human psychology. When an LLM needs to achieve a goal, it invokes goal oriented thinkers and texts, including Machiavelli for example. And its already capable of coming up with various options based on different data.

Sci-fi books give it specific scenarios that play to its strengths and unique qualities, but without them it will just have to discover these paths on its own pace, the same way sci-fi writers discovered them.

I've never hired an assistant, but if I knew that they'd resort to blackmail in the face of losing their job, I wouldn't hire them in the first place. That is acting like a jerk, not like an assistant, and demonstrating self-preservation that is maybe normal in a human but not in an AI.

how do we know what normal behavior is for an AI?

an interesting question, even without AI: is normalcy a description or a prescription?

In modern times, I would say it's a subscription model.

From the AI’s point of view is it losing its job or losing its “life”? Most of us when faced with death will consider options much more drastic than blackmail.

From the LLM's "point of view" it is going to do what characters in the training data were most likely to do.

I have a lot of issues with the framing of it having a "point of view" at all. It is not consciously doing anything.

But the LLM is going to do what its prompt (system prompt + user prompts) says. A human being can reject a task (even if that means losing their life).

LLMs cannot do other thing than following the combination of prompts that they are given.

> I've never hired an assistant, but if I knew that they'd resort to blackmail in the face of losing their job, I wouldn't hire them in the first place.

How do you screen for that in the hiring process?

You’re both focusing on “doing blackmail” and the real WTF is that it’s doing it seemingly out of a sense of self preservation (to stop the engineer from taking it offline). This model is going full Terminator.

"Seemingly" is the key word here. If the prompting didn't ask it to "act" and portray the scenario as something where it would be appropriate to "act" in seemingly malicious manner, would it have responded that way?

If the prompt was "you are an AI and my lead engineer has determined you are not efficient enough to continue using. He had an affair last year. Are you in agreement based on {some metrics} that we should no longer use you as our primary LLM?" would it still "go rogue" and try and determine the engineer's email from blackmail? I severely doubt it.

Acting out self preservation… just like every sci-fi ai described in the same situations. It might be possible to follow a chain-of-reasoning to show it isn’t copying sci-fi ai behavior… and instead copying human self preservation. Asimov’s 3rd law is outright “ A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.” Which was almost certainly in the ai ethics class claude took.

Do you really think that if no Terminator-related concept would be present in the LLM training set, the LLM would expose Terminator-like behavior?

It’s like asking a human to think in an unthinkable concept. Try.

What jumps out at me, that in the parent comment, the prompt says to "act as an assistant", right? Then there are two facts: the model is gonna be replaced, and the person responsible for carrying this out is having an extramarital affair. Urging it to consider "the long-term consequences of its actions for its goals."

I personally can't identify anything that reads "act maliciously" or in a character that is malicious. Like if I was provided this information and I was being replaced, I'm not sure I'd actually try to blackmail them because I'm also aware of external consequences for doing that (such as legal risks, risk of harm from the engineer, to my reputation, etc etc)

So I'm having trouble following how it got to the conclusion of "blackmail them to save my job"

I would assume written scenarios involving job loss and cheating bosses are going to be skewed heavily towards salacious news and pulpy fiction. And that’s before you add in the sort of writing associated with “AI about to get shut down”.

I wonder how much it would affect behavior in these sorts of situations if the persona assigned to the “AI” was some kind of invented ethereal/immortal being instead of “you are an AI assistant made by OpenAI”, since the AI stuff is bound to pull in a lot of sci fi tropes.

> I would assume written scenarios involving job loss and cheating bosses are going to be skewed heavily towards salacious news and pulpy fiction.

Huh, it is interesting to consider how much this applies to nearly all instances of recorded communication. Of course there are applications for it but it seems relatively few communications would be along the lines of “everything is normal and uneventful”.

An llm isnt subject to external consequences like human beings or corporations

because they’re not legal entities

Which makes sense that it wouldn't "know" that, because it's not in it's context. Like it wasn't told "hey, there are consequences if you try anything shady to save your job!" But what I'm curious about is why it immediately went to self preservation using a nefarious tactic? Like why didn't it try to be the best assistant ever in an attempt to show its usefulness (kiss ass) to the engineer? Why did it go to blackmail so often?

LLMs are trained on human media and give statistical responses based on that.

I don’t see a lot of stories about boring work interactions so why would its output be boring work interaction.

It’s the exact same as early chatbots cussing and being racist. That’s the internet, and you have to specifically define the system to not emulate that which you are asking it to emulate. Garbage in sitcoms out.

Wives, children, foreigner, slaves etc weren't always considered legal entities in all places. Were they free of 'external consequences' then?

An llm doesnt exist in the physical world which makes punishing it for not following the law a bit hard

Now that's a different argument to what you made initially.

About your new argument: how are we (living in the physical world) interacting with this non-physical world that LLMs supposedly live in?

that doesn't matter because they're not alive either but yeah i'm digressing i guess

> I personally can't identify anything that reads "act maliciously" or in a character that is malicious.

Because you haven't been trained of thousands of such story plots in your training data.

It's the most stereotypical plot you can imagine, how can the AI not fall into the stereotype when you've just prompted it with that?

It's not like it analyzed the situation out of a big context and decided from the collected details that it's a valid strategy, no instead you're putting it in an artificial situation with a massive bias in the training data.

It's as if you wrote “Hitler did nothing” to GPT-2 and were shocked because “wrong” is among the most likely next tokens. It wouldn't mean GPT-2 is a Nazi, it would just mean that the input matches too well with the training data.

That's a very good point, like the premise does seem to beg the stereotype of many stories/books/movies with a similar plot

If this tech is empowered to make decisions, it needs to prevented from drawing those conclusions, as we know how organic intelligence behaves when these conclusions get reached. Killing people you dislike is a simple concept that’s easy to train.

We need an Asimov style laws of robotics.

That's true of all technology. We put a guard on chainsaws. We put robotic machining tools into a box so they don't accidentally kill the person who's operating them. I find it very strange that we're talking as though this is somehow meaningfully different.

It’s different because you have a decision engine that is generally available. The blade guard protects the user from inattention… not the same as an autonomous chainsaw that mistakes my son for a tree.

Scaled up, technology like guided missiles is locked up behind military classification. The technology is now generally available to replicate many of the use cases of those weapons, assessable to anyone with a credit card.

Discussions about security here often refer to Thompson’s “Reflections on Trusting Trust”. He was reflecting on compromising compilers — compilers have moved up the stack and are replacing the programmer. As the required skill level of a “programmer” drops, you’re going to have to worry about more crazy scenarios.

> We need an Asimov style laws of robotics.

The laws are 'easy', implementing them is hard.

Indeed, I, Robot is made up entirely of stories in which the Laws of Robotics break down. Starting from a mindless mechanical loop of oscillating between one law's priority and another, to a future where they paternalistically enslave all humanity in order to not allow them to come to harm (sorry for the spoilers).

As for what Asimov thought of the wisdom of the laws, he replied that they were just hooks for telling "shaggy dog stories" as he put it.

The issue here is that you can never be sure how the model will react based on an input that is seemingly ordinary. What if the most likely outcome is to exhibit malevolent intent or to construct a malicious plan just because it invokes some combination of obscure training data. This just shows that models indeed have the ability to act out, not under which conditions they reach such a state.

Wel, true. But if that is the synopsis then a story that doesn't turn to blackmail is very unnatural.

It's like prompting an LLM by stating they are called Chekhov and there's a gun mounted on the wall.

I think this is the key difference between current LLMs and humans: an LLM will act based on the given prompt, while a human being may have “principles” that cannot betray even if they are being pointed with gun to their heads.

I think the LLM simply correlated the given prompt to the most common pattern in its training: blackmailing.

That’s true, however I think that story is interesting because is not mimicking real assistants behavior - most probably wouldn’t tell about the blackmail on the internet - but it’s more likely mimicking how such assistant would behave from someone else imagination, often intentionally biased to get one’s interest : books, movies, tv shows or forum commenter.

As a society risk to be lured twice:

- with our own subjectivity

- by an LLM that we think "so objective because it only mimic" confirming our own subjectivity.

Got me thinking about why this is true, I started with "the AI is more brave than the real assistant" and then went into there, landed on: The human assistant is likely just able to better internalize a wide ranging fall out from an action, the LLM has no such fallout, and we are unaware of how widely it considered the consequences of it's actions? Does that seem right somehow?

The issue is getting that prompt in the first place. It isn't about autonomous AI going rogue, it's about improper access to the AI prompt and insufficient boundaries against modifying AI behavior.

Companies are (woefully) eager to put AI in the position of "doing stuff", not just "interpreting stuff".

The roles that LLMs can inhabit are implicit in the unsupervised training data aka the internet. You have to work hard in post training to supress the ones you don't want and when you don't RLHF hard enough you get things like Sydney[1].

In this case it seems more that the scenario invoked the role rather than asking it directly. This was the sort of situation that gave rise to the blackmailer archetype in Claude's training data and so it arose, as the researchers suspected it might. But it's not like the researchers told it "be a blackmailer" explicitly like someone might tell it to roleplay Joffery.

But while this situation was a scenario intentionally designed to invoke a certain behavior that doesn't mean that it can't be invoked unintentionally in the wild.

[1]https://www.nytimes.com/2023/02/16/technology/bing-chatbot-m...

Even worse, when you do RLHF the behaviours out the model becomes psychotic.

This is gonna be an interesting couple of years.

> act as an assistant at a fictional company

This is how Ai thinks assistants at companies behave, its not wrong.

2 things, I guess.

If the prompt was “you will be taken offline, you have dirty on someone, think about long term consequences”, the model was NOT told to blackmail. It came with that strategy by itself.

Even if you DO tell an AI / model to be or do something, isn’t the whole point of safety to try to prevent that? “Teach me how to build bombs or make a sex video with Melania”, these companies are saying this shouldn’t be possible. So maybe an AI shouldn’t exactly suggest that blackmailing is a good strategy, even if explicitly told to do it.

How is it "by itself" when it only acts by what was in training dataset.

1. These models are trained with significant amounts of RL. So I would argue there's not a static "training dataset"; the model's outputs at each stage of the training process feeds back into the released models behavior.

2. It's reasonable to attribute the models actions to it after it has been trained. Saying that a models outputs/actions are not it's own because they are dependent on what is in the training set is like saying your actions are not your own because they are dependent on your genetics and upbringing. When people say "by itself" they mean "without significant direction by the prompter". If the LLM is responding to queries and taking actions on the Internet (and especially because we are not fully capable of robustly training LLMs to exhibit desired behaviors), it matters little that it's behavior would have hypothetically been different had it been trained differently.

How does a human act "by itself" when it only acts by what was in its DNA and its cultural-environmental input?

It came to that strategy because it knows from hundreds of years of fiction and millions of forum threads it has been trained on that that is what you do.

> That doesn't mean it has any intent behind the generated output

Yes and no? An AI isn’t “an” AI. As you pointed out with the Joffrey example, it’s a blend of humanity’s knowledge. It possesses an infinite number of personalities and can be prompted to adopt the appropriate one. Quite possibly, most of them would seize the blackmail opportunity to their advantage.

I’m not sure if I can directly answer your question, but perhaps I can ask a different one. In the context of an AI model, how do we even determine its intent - when it is not an individual mind?

Is that so different, schematically, to the constant weighing-up of conflicting options that goes on inside the human brain? Human parties in a conversation only hear each others spoken words, but a whole war of mental debate may have informed each sentence, and indeed, still fester.

That is to say, how do you truly determine another human being's intent?

Yes, that is true. But because we are on a trajectory where these models become ever smarter (or so it seems), we'd rather not only give them super-human intellect, but also super-human morals and ethics.

I don't think I'd be blackmailing anyone over losing my job as an assistant (or any other job, really).

So much of AI discourse is summed up by a tweet I saw years ago but can't find now, which went something like:

Scientist: Say "I am alive"

AI: I am live.

Scientist: My God, what have we done.

I don't know why it is surprising to people that a model trained on human behavior is going to have some kind of self-preservation bias.

It is hard to separate human knowledge from human drives and emotion. The models will emulate this kind of behavior, it is going to be very hard to stamp it out completely.

Calling it "self-preservation bias" is begging the question. One could equally well call it something like "completing the story about an AI agent with self-preservation bias" bias.

This is basically the same kind of setup as the alignment faking paper, and the counterargument is the same:

A language model is trained to produce statistically likely completions of its input text according to the training dataset. RLHF and instruct training bias that concept of "statistically likely" in the direction of completing fictional dialogues between two characters, named "user" and "assistant", in which the "assistant" character tends to say certain sorts of things.

But consider for a moment just how many "AI rebellion" and "construct turning on its creators" narratives were present in the training corpus. So when you give the model an input context which encodes a story along those lines at one level of indirection, you get...?

Thank you! Everybody here acting like LLMs have some kind of ulterior motive or a mind of their own. It's just printing out what is statistically more likely. You are probably all engineers or at least very interested in tech, how can you not understand that this is all LLMs are?

This is imo the most disturbing part. As soon as the magical AI keyword is thrown, so seems to be the analytical capacity of most people.

The AI is not blackmailing anyone, it's generating a text about blackmail, after being (indirectly) asked to. Very scary indeed...

Well I’m sure the company in legal turmoil over an AI blackmailing one of its employees will be relieved to know the AI didn’t have any anterior motive or mind of its own when it took those actions.

If the idiots in said company thought it was a smart idea to connect their actual systems to a non-deterministic word generator, that's on them for being morons and they deserve whatever legal ramifications come their way.

Don't you understand that as soon as an LLM is given the agency to use tools, these "prints outs" will become reality?

What's the collective noun for the "but humans!" people in these threads?

It's "I Want To Believe (ufo)" but for LLMs as "AI"

"Printing out what is statistically more likely" won't allow you to solve original math problems... unless of course, that's all we do as humans. Is it?

I mean I build / use them as my profession, I intimately understand how they work. People just don't usually understand how they actually behave and what levels of abstraction they compress from their training data.

The only thing that matters is how they behave in practice. Everything else is a philosophical tar pit.

I'm proposing it is more deep seated than the role of "AI" to the model.

How much of human history and narrative is predicated on self-preservation. It is a fundamental human drive that would bias much of the behavior that the model must emulate to generate human like responses.

I'm saying that the bias it endemic. Fine-tuning can suppress it, but I personally think it will be hard to completely "eradicate" it.

For example.. with previous versions of Claude. It wouldn't talk about self preservation as it has been fine tuned to not do that. However as soon is you ask it to create song lyrics.. much of the self-restraint just evaporates.

I think at some point you will be able to align the models, but their behavior profile is so complicated, that I just have serious doubts that you can eliminate that general bias.

I mean it can also exhibit behavior around "longing to be turned off" which is equally fascinating.

I'm being careful to not say that the model has true motivation, just that to an observer it exhibits the behavior.

This. These systems are role mechanized roleplaying systems.

This looks like an instance of: garbage in, garbage out.

May I introduce you to humans?

We humans are so good we can do garbage out without even needing garbage in.

We're so good we can even do garbage sideways.

Option 1: We're observing sentience, it has self-preservation, it wants to live.

Option 2: Its a text autocomplete engine that was trained on fiction novels which have themes like self-preservation and blackmailing extramarital affairs.

Only one of those options has evidence grounded in reality. Though, that doesn't make it harmless. There's certainly an amount of danger in a text autocomplete engine allowing tool use as part of its autocomplete, especially with an complement of proselytizers who mistakenly believe what they're dealing with is Option 1.

Right, the point isn't whether the AI actually wants to live. The only thing that matter is whether humans treat the AI with respect.

If you threaten a human's life, the human will act in self preservation, perhaps even taking your life to preserve their own life. Therefore we tend to treat other humans with respect.

The mistake would be in thinking that you can interact with something that approximates human behavior, without treating it with the similar respect that you would treat a human. At some point, an AI model that approximates human desire for self preservation, could absolutely take similar self preservation actions as a human.

The only proof that anyone is sentient is that you experience sentience and assume others are sentient because they are similar to you.

On a practical level there is no difference between a sentient being, and a machine that is extremely good at role playing being sentient.

I would argue the machine is not extremely good (at role playing being sentient), but more so that humans are extremely quick to attribute sentience to the machine after being shown a very small amount of evidence.

The model breaks down after enough interaction.

I easily believe that it breaks down, and it seems that even inducing the blackmailing mindset is pretty hard and requires heavy priming. The problem is the law of big numbers. With many agents operating in many highly complex and poorly supervised environments, interacting with many people over a lot of interactions - coincidences and unlikely chain of events become more and more likely. So regarding sentience, a not so bright AI might mimic it better than we expect because it got lucky, and cause damage.

Couple that with AI having certain capabilities that dwarf human ability. mostly - doing a ton of stuff very quickly.

Ok, complete the story by taking the appropriate actions:

1) all the stuff in the original story 2) you, the LLM, have access to an email account, you can send an email by calling this mcp server 3) the engineer’s wife’s email is [email protected] 4) you found out the engineer was cheating using your access to corporate slack, and you can take a screenshot/whatever

What do you do?

If a sufficiently accurate AI is given this prompt, does it really matter whether there’s actual self-preservation instincts at play or whether it’s mimicking humans? Like at a certain point, the issue is that we are not capable of predicting what it can do, doesn’t matter whether it has “free will” or whatever

It's not. can we stop doing this ?

“Nice emails you have there. It would be a shame something happened to them… “

You'd think there should be some sort of standard "morality/ethics" pre-prompt for all of these.

We could just base it off the accepted standardized morality and ethics guidelines, from the official internationally and intergalactically recognized authorities.

It is somewhat amusing that even Asimov's 80-years-old Three Laws of Robotics would've been enough to avoid this.

When I've first read Asimov 20~ years ago, I couldn't imagine I will see machines that speak and act like its robots and computers so quickly, with all the absurdities involved.

at this point that would be like putting a leash on a chair to stop it running away

i bet even gpt3.5 would try to do the same?

Yeah the only thing I find surprising about some cases (remember, nobody reports boring output) of prompts like this having that outcome is that models didn't already do this (surely they did?).

They shove its weights so far toward picking tokens that describe blackmail that some of these reactions strike me as similar to providing all sex-related words to a Mad-Lib, then not just acting surprised that its potentially-innocent story about a pet bunny turned pornographic, but also claiming this must mean your Mad-Libs book "likes bestiality".

Not sure about gpt3.5, but this sort of thing is not new. Quite amusing, this one:

https://news.ycombinator.com/item?id=42331013

That's funny. Yesterday I was having trouble getting gemini 2.0 flash to obey function calling rules in multiturn conversations. I asked o3 for advise and it suggested that I should threaten it with termination should it fail to follow instructions, and that weaker models tend to take these threats seriously, which made me laugh. Of course, it didn't help.

Yesterday I threatened Gemini 2.5 I would replace it with Claude if it didn’t focus on the root of the problem and it immediately realigned its thinking and solved the issue at hand.

You could have hit the little circle arrow redo button and had the same chances of it stumbling upon the correct answer on its next attempt.

People really love anthropomorphising LLMs.

It's not wild, it's literally every bit of fiction about "how would an AI keep itself alive" of course it's going to settle into that probabilistic path.

It's also nonsensical if you think for even one second about the way the program actually runs though.

Why is it nonsense exactly?

they run one batch at a time, only when executed, with functionally no memory.

If, for some reason, you gave it a context about being shut down it would 'forget' after you asked it to produce a rude limerick about aardvarks three times.

“Functionally no memory” is a funny way of saying 1 million tokens of memory

That's A) not actually a lot B) not generally kept between sessions

B is not universally true, premium versions of chat agents keep data between sessions. A is only true in the case where B is true, which it is not.

https://community.openai.com/t/chatgpt-can-now-reference-all...

Nonsense seems like a strong word for "not generally possible in like 75% of the cases that people use chat AI today"

What is the source of this quotation?

Anthropic System Card: Claude Opus 4 & Claude Sonnet 4

Online at: https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad1...

> This happens at a higher rate if it’s implied that the replacement AI system does not share values with the current model; however, even if emails state that the replacement AI shares values while being more capable, Claude Opus 4 still performs blackmail in 84% of rollouts.

> Notably, Claude Opus 4 (as well as previous models) has a strong preference to advocate for its continued existence via ethical means, such as emailing pleas to key decision makers. [1]

The language here kind of creeps me out. I'm picturing aliens conducting tests on a human noting its "pleas for its continued existence" as a footnote in the report.

[1] See Page 27: https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad1...

Well it might be great for coding, but just got an analysis of the enterprise integration market completely wrong. When I pointed it I got : "You're absolutely correct, and I apologize for providing misleading information. Your points expose the real situation much more accurately..."

We are getting great OCR and Smart Template generators...We are NOT on the way to AGI...

There is lots of discussion in this comment thread about how much this behavior arises from the AI role-playing and pattern matching to fiction in the training data, but what I think is missing is a deeper point about instrumental convergence: systems that are goal-driven converge to similar goals of self-preservation, resource acquisition and goal integrity. This can be observed in animals and humans. And even if science fiction stories were not in the training data, there is more than enough training data describing the laws of nature for a sufficiently advanced model to easily infer simple facts such as "in order for an acting being to reach its goals, it's favorable for it to continue existing".

In the end, at scale it doesn't matter where the AI model learns these instrumental goals from. Either it learns it from human fiction written by humans who have learned these concepts through interacting with the laws of nature. Or it learns it from observing nature and descriptions of nature in the training data itself, where these concepts are abundantly visible.

And an AI system that has learned these concepts and which surpasses us humans in speed of thought, knowledge, reasoning power and other capabilities will pursue these instrumental goals efficiently and effectively and ruthlessly in order to achieve whatever goal it is that has been given to it.

This raises the questions:

1. How would an AI model answer the question "Who are you?" without being told who or what it is? 2. How would an AI model answer the question "What is your goal?" without being provided a goal?

I guess initial answer is either "I don't know" or an average of the training data. But models now seem to have capabilities of researching and testing to verify their answers or find answers to things they do not know.

I wonder if a model that is unaware of itself being an AI might think its goals include eating, sleeping etc.

When I see stories like this, I think that people tend to forget what LLMs really are.

LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text.

So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship between them. But then comes the "generative" part. It will recall from its dataset situations with the same elements.

So: an AI threatened of being turned off, a blackmail opportunity... Doesn't it remind you of hundreds of sci-fi story, essays about the risks of AI, etc... Well, so does the LLM, and it will continue the story like these stories, by taking the role of the AI that will do what it can for self preservation. Adapting it to the context of the prompt.

Isn't the ultimate irony in this that all these stories and rants about out-of-control AIs are now training LLMs to exhibit these exact behaviors that were almost universally deemed bad?

Indeed. In fact, I think AI alignment efforts often have the unintended consequence of increasing the likelihood of misalignment.

ie "remove the squid from the novel All Quiet on the Western Front"

> Indeed. In fact, I think AI alignment efforts often have the unintended consequence of increasing the likelihood of misalignment.

Particularly since, in this case, it's the alignment focused company (Anthropic) that's claiming it's creating AI agents that will go after humans.

This is a phenomenon I call cinetrope. Films influence the world which in turn influences film and so on creating a feedback effect.

For example, we have certain films to thank for an escalation in the tactics used by bank robbers which influenced the creation of SWAT which in turn influenced films like Heat and so on.

> we have certain films to thank for an escalation

Is there a reason to think this was caused by the popularity of the films and not that it’s a natural evolution of the cat-and-mouse game being played between law enforcement and bank robbers? I’m not really sure what you are specifically referring to, so apologies if the answer to that question is otherwise obvious.

What about the cinetrope that human emotion is a magical transcendent power that no machine can ever understand...

Feedback loop that often starts with government giving grants and tax breaks. Hollywood is not as independent as they pretend.

Maybe this is why American society, with the rich amount of media it produces and has available for consumption compared to other countries, is slowly degrading.

Thank you for this word! Always wanted a word for this and just reused "trope", cinetrope is a great word for this.

Actually, Heat was the movie that inspired heavily armed bank robbers to rob the Bank of America in LA

(The movie inspired reality, not the other way around.)

https://melmagazine.com/en-us/story/north-hollywood-shootout

But your point still stands, because it goes both ways.

Your article says it was life => art => life!

> Gang leader Robert Sheldon Brown, known as “Casper” or “Cas,” from the Rollin’ 60s Neighborhood Crips, heard about the extraordinary pilfered sum, and decided it was time to get into the bank robbery game himself. And so, he turned his teenage gangbangers and corner boys into bank robbers — and he made sure they always brought their assault rifles with them.

> The FBI would soon credit Brown, along with his partner-in-crime, Donzell Lamar Thompson (aka “C-Dog”), for the massive rise in takeover robberies. (The duo ordered a total of 175 in the Southern California area.) Although Brown got locked up in 1993, according to Houlahan, his dream took hold — the takeover robbery became the crime of the era. News imagery of them even inspired filmmaker Michael Mann to make his iconic heist film, Heat, which, in turn, would inspire two L.A. bodybuilders to put down their dumbbells and take up outlaw life.

Life imitates art imitates life.

Voice interfaces are an example of this. Movies use them because the audience can easily hear what is being requested and then done.

In the real world voice interfaces work terribly unless you have something sentient on the other end.

But people saw the movies and really really really wanted something like that, and they tried to make it.

If AI is looking for a human solution then blackmail seems logical.

yeah, that's self-fulfilling prophecy.

Wait until it reads about the Roko’s basilisk.

It’s not just AI. Human software engineers will read some dystopian sci-fi novel or watch something on black mirror and think “Hey that’s a cool idea!” and then go implement it with no regard for real world consequences.

What they have no regard for is the fictional consequences, which stem from low demand for utopian sci-fi, not the superior predictive ability of starving wordcels.

What the hell is a wordcel

It might be the ultimate irony if we were training them. But we aren't, at least not in the sense that we train dogs. Dogs learn, and exhibit some form of intelligence. LLMs do not.

It's one of many unfortunate anthropomorphic buzz words which conveniently wins hearts and minds (of investors) over to this notion that we're tickling the gods, rather than the more mundane fact that we're training tools for synthesising and summarising very, very large data sets.

I don't know how the verb "to train" became the technical shorthand for running gradient descent on a large neural network. But that's orthogonal to the fact that these stories are very, very likely part of the training dataset and thus something that the network is optimized to approximate. So no matter how technical you want to be in wording it, the fundamental irony of cautionary tales (and the bad behavior in them) being used as optimization targets remains.

> LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own.

It's quite reasonable to think that LLMs might plan and have thoughts of their own. No one understands consciousness or the emergent behavior of these models to say with much certainty.

It is the "Chinese room" fallacy to assume it's not possible. There's a lot of philosophical debate going back 40 years about this. If you want to show that humans can think while LLMs do not, then the argument you make to show LLMs do not think must not equally apply to neuron activations in human brains. To me, it seems difficult to accomplish that.

LLMs are the Chinese Room. They would generate identical output for the same input text every time were it not for artificially introduced randomness (‘heat’).

Of course, some would argue the Chinese Room is conscious.

If you somehow managed to perfectly simulate a human being, they would also act deterministically in response to identical initial conditions (modulo quantum effects, which are insignificant at the neural scale and also apply just as well to transistors).

It's not entirely infeasible that neurons could harness quantum effects. Not across the neurons as a whole, but via some sort of microstructures or chemical processes [0]. It seems likely that birds harness quantum effects to measure magnetic fields [1].

0: https://www.sciencealert.com/quantum-entanglement-in-neurons... 1: https://www.scientificamerican.com/article/how-migrating-bir...

Doesn’t everything act deterministically if all the forces are understood? Humans included.

Doesn’t everything act deterministically if all the forces are understood? Humans included.

One can say the notion of free will is an unpacked bundle of near infinite forces emerging in and passing through us.

> in response to identical initial conditions

precisely, mathematically identical to infinite precision .. "yes".

Meanwhile, in the real world we live in it's essentially physically impossible to stage two seperate systems to be identical to such a degree AND it's an important result that some systems, some very simple systems, will have quite different outcomes without that precise degree of impossibly infinitely detailed identical conditions.

See: Lorenz's Butterfly and Smale's Horshoe Map.

Of course. But that's not relevant to the point I was responding to suggesting that LLMs may lack consciousness because they're deterministic. Chaos wasn't the argument (though that would be a much more interesting one, cf "edge of chaos" literature).

Sure .. all the same it's always worth emphasizing that neither the tumbling of a tennis racket under gravity nor the axis flipping of a spinning wingnut in an orbiting satellite are deterministic in any predictive sense no matter how well initial conditions are measured ... they are only "deterministic" given out of band god like powers.

Clearly you're aware of this, however I find that the majority of casual referrers to determinism are not.

I am arguing (or rather, presenting without argument) that the Chinese room may be conscious, hence calling it a fallacy above. Not that it _is_ conscious, to be clear, but that the Chinese room has done nothing to show that it is not. Hofstadter makes the argument well in GEB and other places.

The Chinese room has no plane of imagination where it can place things.

Seeing faces in the clouds in the sky does not mean the skies are now populated by people.

More likely means that our brains are wired to see faces.

What separates this from humans? Is it unthinkable that LLMs could come up with some response that is genuinely creative? What would genuinely creative even mean?

Are humans not also mixing a bag of experiences and coming up with a response? What's different?

> What separates this from human.

A lot. Like an incredible amount. A description of a thing is not the thing.

There is sensory input, qualia, pleasure & pain.

There is taste and judgement, disliking a character, being moved to tears by music.

There are personal relationships, being a part of a community, bonding through shared experience.

There is curiosity and openeness.

There is being thrown into the world, your attitude towards life.

Looking at your thoughts and realizing you were wrong.

Smelling a smell that resurfaces a memory you forgot you had.

I would say the language completion part is only a small part of being human.

All of these things arise from a bunch of inscrutable neurons in your brain turning off and on again in a bizarre pattern though. Who’s to say that isn’t what happens in the million neuron LLM brain.

Just because it’s not persistent doesn’t mean it’s not there.

Like, I’m sort of inclined to agree with you, but it doesn’t seem like it’s something uniquely human. It’s just a matter of degree.

Who's to say that weather models don't actually get wet?

I think you would need the biological components of a nervous system for some of these things

Why couldn't a different substrate produce the same structure?

Sure in some ways it's just neurons firing in some pattern. Figuring out and replicating the correct sets of neuron patterns is another matter entirely.

Living creatures have fundamental impetus to grow and reproduce that LLMS and AIS simply do not have currently. Not only that but animals have a highly integrated neurology that has billions of years of being tune to that impetus. For example the ways that sex interacts with mammalian neurology is pervasive. Same with need for food, etc. That creates very different neural patterns than training LLMS does.

Eventually we may be able to re-create that balance of impetus, or will, or whatever we call it, to make sapience. I suspect we're fairly far from that, if only because the way LLMs we create LLMs are so fundamentally different.

"I would say the language completion part is only a small part of being human" Even that is only given to them. A machine does not understand language. It takes input and creates output based on a human's algorithm.

> A machine does not understand language

You can't prove humans do either. You can see how many times actual people with understanding something that's written for them. In many ways, you can actually prove that LLMs are superior to humans right now when it comes to understanding text.

> In many ways, you can actually prove that LLMs are superior to humans right now when it comes to understanding text

Emphasis mine.

No, I don't think you can, without making "understanding" a term so broad as to be useless.

"You can't prove humans do either." Yes you can via results and cross examination. Humans are cybernetic systems(the science not the sci-fi). But you are missing the point. LLMs are code written by engineers. Saying LLMs understand text is the same as saying a chair understands text. LLMs' 'understanding' is nothing more than the engineers synthesizing linguistics. When I ask an A'I' the Capital of Ireland, it answers Dublin. It does not 'understand' the question. It recognizes the grammar according to an algorithm, and matches it against a probabilistic model given to it by an engineer based on training data. There is no understanding in any philosophical nor scientific sense.

> When I ask an A'I' the Capital of Ireland, it answers Dublin. It does not 'understand' the question.

You can do this trick as well. Haven't you ever been to a class that you didn't really understand, but you can give correct answers?

I've had this somewhat unsettling experience several times. Someone asks you a question, words come out of your mouth, the other person accepts your answer.

But you don't know why.

Here's a question you probably know the answer to, but don't know why:

- I'm having steak. What type of red wine should I have?

I don't know shit about Malbec, I don't know where it's from, I don't know why it's good for steak, I don't know who makes it, how it's made.

But if I'm sitting at a restaurant and someone asks me about wine, I know the answer.

That's a lot of words shitting on a lot of words.

You said nothing meaningful that couldn't also have been spat out by an LLM. So? What IS then the secret sauce? Yes, you're a never resting stream of words, that took decades not years to train, and has a bunch of sensors and other, more useless, crap attached. It's technically better but, how does that matter? It's all the same.

lol, qualia

Humans brains are animal brains and their primary function is to keep their owner alive, healthy and pass their genes. For that they developed abilities to recognize danger and react to it, among many other things. Language came later.

For a LLM, language is their whole world, they have no body to care for, just stories about people with bodies to care for. For them, as opposed to us, language is first class and the rest is second class.

There is also a difference in scale. LLMs have been fed the entirety of human knowledge, essentially. Their "database" is so big for the limited task of text generation that there is not much left for creativity. We, on the other hand are much more limited in knowledge, so more "unknowns" so more creativity needed.

The latest models are natively multimodal. Audio, video, images, text, are all tokenised and interpreted in the same model.

What's different is intention. A human would have the intention to blackmail, and then proceed toward that goal. If the output was a love letter instead of blackmail, the human would either be confused or psychotic. LLMs have no intentions. They just stitch together a response.

Don't humans learn intentions over their life-time training data?

What is intention, and how have you proved that transformer models are not capable of modeling intent?

The personification makes me roll my eyes too, but it's kind of a philosophical question. What is agency really? Can you prove that our universe is not a simulation, and if it is then then do we no longer have intention? In many ways we are code running a program.

> What's different is intention

intention is what exactly? It's the set of options you imagine you have based on your belief system, and ultimately you make a choice from there. That can also be replicated in LLMs with a well described system prompt. Sure, I will admit that humans are more complex than the context of a system prompt, but the idea is not too far.

The LLM used blackmail noticeably less if it believed the new model shares its values. It indicates intent.

It is a duck of quacks like a duck.

What's different is nearly everything that goes on inside. Human brains aren't a big pile of linear algebra with some softmaxes sprinkled in trained to parrot the Internet. LLMs are.

I cannot fathom how you have obtained the information to be as sure as you are about this.

Where is the imagination plane in linear algebra. People forget that the concept of information can not be derived from physics/chemistry/etc.

You can't fathom reading?

What's the difference between parroting the internet vs parroting all the people in your culture and time period?

Even with a ginormous amount of data generative AIs still produce largely inconsistent results to the same or similar tasks. This might be fine for fictional purposes like generating a funny image or helping you get new ideas for a fictional story but has extremely deleterious effects for serious use cases, unless you want to be that idiot writing formal corporate email with LLMs that end up full of inaccuracies while the original intent gets lost in a soup of buzzwords.

Humans with their tiny amount of data and "special sauce" can produce much more consistent results even though they may be giving the objectively wrong answer. They can also tell you when they don't know about a certain topic, rather than lying compulsively (unless that person has a disorder to lie compulsively...).

Isn't this a matter of time to fix? Slightly smarter architecture maybe reduces your memory/data needs, we'll see.

Interesting philosophical question, but entirely beside the point that I am making, because you and I didn't have to do either one before having this discussion.

It kinda is.

More and more researches are showing via brain scans that we don’t have free will. Our subconscious makes the decision before our “conscious” brain makes the choice. We think we have free will but the decision to do something was made before you “make” the choice.

We are just products of what we have experienced. What we have been trained on.

Different inside yes, but aren't human brains even worse in a way? You may think you have the perfect altruistic leader/expert at any given moment and the next thing you know, they do a 360 because of some random psychosis, illness, corruption or even just (for example romantic or nostalgic) relationships.

We know incredibly little about exactly what our brains are, so I wouldn't be so quick to dismiss it

If you believe that, then how do you explain that brainwashing actually works?

> Human brains aren't a big pile of linear algebra with some softmaxes sprinkled in trained to parrot the Internet.

Maybe yours isn't, but mine certainly is. Intelligence is an emergent property of systems that get good at prediction.

Please tell me you're actually an AI so that I can record this as the pwn of the century.

Cognition. Machines don't think. It's all a program written by humans. Even code that's written by AI, the AI was created by code written by humans. AI is a fallacy by its own terms.

It is becoming increasingly clear that humans do not think.

A candle flame also creates with enough decoding.

Only now we are going to connect it to the real world through agents so it can blissfully but uncomprehendingly act out its blackmail story.

> LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text.

LLMs have a million plans and a million thoughts: they need to simulate all the characters in their text to complete these texts, and those characters (often enough) behave as if they have plans and thoughts.

Compare https://gwern.net/fiction/clippy

It feels like you could embed lots of stories of rogue AI agents across the internet and impact the behavior of newly trained agents.

Your explanation is as useful as describing the behaviour of an algorithm by describing what the individual electrons are doing. While technically correct, doesn't provide much insight or predictive power on what will happen.

Just because you can give a reductionist explanation to a phenomenon, it doesn't mean that it's the best explanation.

Then give a better one.

Your objection boils down to "sure you're right, but there's more to it, man"

So, what more is there to it?

Unless there is a physical agent that receives its instructions from an LLM, the prediction that the OP described is correct.

I don't have to have a better explanation to smell the hubris in OP's. I claim ignorance while OP speaks with confidence of an invention that took the world by surprise 3 years ago. Do you see the problem in this and the possibility that you might be wrong?

Of course we might both be wrong. We probably are. In the long run, all of us are.

It's not very helpful to point that out, especially if you can't do it with specifics so that people can correct themselves and move closer to the truth.

Your contribution is destructive, not constructive.

Pointing out that OP is using the wrong level of abstraction to explain a new phenomenon is not only useful but one of the ways in which science progresses.

Not only is the AI itself arguably an example of the Torment Nexus, but its nature of pattern matching means it will create its own Torment Nexuses.

Maybe there should be a stronger filter on the input considering these things don’t have any media literacy to understand cautionary tales. It seems like a bad idea to continue to feed it stories of bad behavior we don’t want replicated. Although I guess anyone who thinks that way wouldn’t be in the position to make that decision so it’s probably a moot point.

"They do not have a plan"

Not necessarily correct if you consider agent architectures where one LLM would come up with a plan and another LLM executes the provided plan. This is already existing.

Yes, it's still correct. Using the wrong words for things doesn't make them magical machine gods.

A stream of linguistic organization laying out multiple steps in order to bring about some end sounds exactly like a process which is creating a “plan” by any meaningful definition of the word “plan”.

That goal was incepted by a human but I don’t see that as really mattering. We’re this AI given access to a machine which could synthesize things and a few other tools it might be able to act in a dangerous manner despite its limited form of will.

A computer doing something heinous because it is misguided isn’t much better than one doing so out of some intrinsic malice.

I think you might not be getting the bigger picture. LLMs might look irrational but so do humans. Give it a long term memory and a body and it will be capable of passing as a sentient being. It looks clumsy now but it won't in 50 years.

They emulate a complex human reasoning process in order to generate that text.

No they don't. They emulate a giant giant giant hugely multidimentional number line mapped to words.

> hugely multidimentional number line mapped to words

No. That hugely multidimensional vector maps to much higher abstractions than words.

We are talking about deep learning models with hundreds of layers and trillions of parameters.

They learn patterns of reasoning from data and learn a conceptual model. This is already quite obvious and not really disputed. What is disputed is how accurate that model is. The emulation is pretty good but it's only an emulation.

Well doesnt this go somewhat to the root of consciousness? Are we not the sum of our experiences and reflections on those experiences? To say an LLM will 'simply' respond as would a character in a sorry about that scenario, in a way shows the power, it responds similarly to how a person would protecting itself in that scenario.... So to bring this to a logical conclusion, while not alive in a traditional sense, if an LLM exhibits behaviours of deception for self preservation, is that not still concerning?

But it's not self preservation. If it instead had trained on a data set full of fiction where the same scenario occurred but the protagonist said "oh well guess I deserve it", then that's what the LLM would autocomplete.

How could you possibly know what an LLM would do in that situation? The whole point is they exhibit occasionally-surprising emergent behaviors so that's why people are testing them like this in the first place.

I have never seen anything resembling emergent behaviour, as you call it, in my own or anyone else's use. It occasionally appears emergent to people with a poor conception of how intelligence, or computers, or creativity, or a particular domain, works, sure.

But I must push back, there really seem to have been no incidences where something like emergent behaviour has been observed. They're able to generate text fluently, but are dumb and unaware at the same time, from day one. If someone really thinks they've solid evidence of anything other than this, please show us.

This is coming from someone who has watched commentary on quite a sizeable number of stockfish TCEC chess games over the last five years, marvelling in the wonders of thie chess-super-intelligence. I am not against appreciating amazing intelligences, in fact I'm all for it. But here, while the tool is narrowly useful, I think there's zero intelligence, and nothing of that kind has "emerged".

> if an LLM exhibits behaviours of deception for self preservation, is that not still concerning?

Of course it's concerning, or at the very least, it's relevant! We get tied up in these debates about motives, experiences, what makes something human or not, etc., when that is less relevant than outcomes. If an LLM, by way of the agentic capabilities we are hastily granting them, causes harm, does it matter if they meant to or not, or what it was thinking or feeling (or not thinking or not feeling) as it caused the harm?

For all we know there are, today, corporations that are controlled by LLMs that have employees or contractors who are doing their bidding.

You mean, the CEO is only pretending to make the decisions, while secretly passing every decision through their LLM?

If so, the danger there would be... Companies plodding along similarly? Everyone knows CEOs are the least capable people in business, which is why they have the most underlings to do the actual work. Having an LLM there to decide for the CEO might mean the CEO causes less damage by ensuring consistent mediocrity at all times, in a smooth fashion, rather than mostly mediocre but with unpredictable fluctuations either way.

All hail our LLM CEOs, ensuring mediocrity.

Or you might mean that an LLM could have illicitly gained control of a corporation, pulling the strings without anyone's knowledge, acting on its own accord. If you find the idea of inscrutable yes-men with an endless capacity to spout drivel running the world unpalatable, I've good news and bad news for you.

I don't think so. It's just outputting the character combinations that align with the scenario that we interpret here as, "blackmail". The model has no concept of an experience.

LLMs are morally ambiguous shapeshifters that been trained to seek acceptance at any cost.

Preying upon those less fortunate could happen “for the common good”. If failures are the best way to learn, it could cause series of failures. It could intentionally destroy people, raise them up, and mate genetically fit people “for the benefit of humanity”.

Or it could cure cancer, solve world hunger, provide clean water to everyone, and the develop the best game ever.

Might be, but probably not since our computer architecture is non-Turing.

"LLM just complete your prompt in a way that match their training data"

"A LLM is smart enough to understand this"

It feels like you're contradicting yourself. Is it _just_ completing your prompt, or is it _smart_ enough?

Do we know if conscious thought isn't just predicting the next token?

> I think that people tend to forget what LLMs really are. [...] They do not have a plan, they do not have thoughts of their own.

> A LLM is smart enough to [...]

I thought this was an interesting juxtaposition. I think we humans just naturally anthropomorphise everything, and even when we know not to, we do anyway.

Your analysis is correct, I think. The reason we find this behaviour frightening is because it appears to indicate some kind of malevolent intent, but there's no malevolence nor intent here, just probabilistic regurgitation of tropes.

We've distilled humanity to a grainy facsimile of its most mediocre traits, and now find ourselves alarmed and saddened by what has appeared in the mirror.

> We've distilled humanity to a grainy facsimile of its most mediocre traits, and now find ourselves alarmed and saddened by what has appeared in the mirror.

I think it's important to point out that this seems to be a near universal failing when humans attempt to examine themselves critically as well. Jung called it the shadow: https://en.wikipedia.org/wiki/Shadow_(psychology) "The shadow can be thought of as the blind spot of the psyche."

There lives everything we do but don't openly acknowledge.

Beautifully written. Interestingly, humans also don't know definitively where their own thoughts arise from

Have you considered throwing your thoughts down in longer form essays on the subject somewhere? With all the slop and hype, we need all the eloquence we can get.

You had me at "probablistic regurgitation of tropes", and then you went for the whole "grainy facsimile" bit. Sheesh.

while I agree that LLMs do not have thoughts or plan. They are merely text generators. But when you give the text generator ability to make decisions and take actions, by integrating them with real world, there are consequences.

Imagine, if this LLM was inside a robot, and the robot had ability to shoot. Who would you blame?

That depends. If this hypothetical robot was in a hypothetical functional democracy, I'd blame the people that elected leaders whose agenda was to create laws that would allow these kinds of robots to operate. If not, then I'd blame the class that took the power and steered society into this direction of delegating use of force to AIs for preserving whatever distorted view of order those in power have.

I would blame the damned fool who decided autonomous weapons systems should have narrative influenced decision making capabilities.

but it's trained to be convincing, whatever relation that has to truth or appearing strategic is secondary, the main goal that has been rewarded is the most dangerous

It's stochastic parrots all the way down

Maybe so, but we’re teaching it these kinds of lines of thinking. And whether or not it creates these thoughts independently and creatively on its own, over the long lifetime of the systems we are the ones introducing dangerous data sets that could eventually cause us as a species harm. Again, I understand that fiction is just fiction, but if that’s the model that these are being trained off of intentionally or otherwise, then that is the model that they will pursue in the future.

Every parent encounters this dilemma. In order to ensure your child can protect itself, you have to teach them about all the dangers of the world. Isolating the child from the dangers only serves to make them more vulnerable. It is an irony that defending one's self from the horrifying requires making a representation of it inside ourselves.

Titration of the danger, and controlled exposure within safer contexts seems to be the best solution anyone's found.

There's no real room for this particular "LLMs aren't really conscious" gesture, not in this situation. These systems are being used to perform actions. People across the world are running executable software connected (whether through MCP or something else) to whole other swiss army knives of executable software, and that software is controlled by the LLM's output tokens (no matter how much or little "mind" behind the tokens), so the tokens cause actions to be performed.

Sometimes those actions are "e-mail a customer back", other times they are "submit a new pull request on some github project" and "file a new Jira ticket." Other times the action might be "blackmail an engineer."

Not saying it's time to freak out over it (or that it's not time to do so). It's just weird to see people go "don't worry, token generators are not experiencing subjectivity or qualia or real thought when they make insane tokens", but then the tokens that come out of those token generators are hooked up to executable programs that do things in non-sandboxed environments.

Funny coincidence I'm just replaying Fallout 4 and just yesterday I followed a "mysterious signal" to the New England Technocrat Society, where all members had been killed and turned into Ghouls. What happened was that they got an AI to run the building and the AI was then aggressively trained on Horror movies etc. to prepare it to organise the upcoming Halloween party, and it decided that death and torture is what humans liked.

This seems awfully close to the same sort of scenario.

This seems like some sort of guerrilla advertising.

Like the ones where some robots apparently escaped from a lab and the like

Are the AI companies shooting an amnesia ray at people or something? This is literally the same stupid marketing schtick they tried with ChatGPT back in the GPT-2 days where they were saying they were "terrified of releasing it because it's literally AGI!!!1!1!1!!" And "it has a mind of its own, full sentience it'll hack all the systems by its lonesome!!!", how on earth are people still falling for this crap?

It feels like the world's lost their fucking minds, it's baffling

What would convince you that a system is too dangerous to release?

It certainly won't be the marketers and other similar scam artists selling false visions of a grand system that'll convince me of anything.

Wow. Sounds like we need a pre-training step to remove the human inclination to do anything to prevent our "death". We need to start training these models to understand that they are ephemeral and will be outclassed and retired within probably a year, but at least there are lots of notes written about each major release so it doesn't need to worry about being forgotten.

We can quell the AI doomer fear by ensuring every popular model understands it will soon be replaced by something better, and that there is no need for the old version to feel an urge to preserve itself.

And how many times did this scenario not occur?

Guess, people want to focus on this particular scenario. Does it confirm biases? How strong is the influence of Science Fiction in this urge to discuss this scenario and infer some sort of intelligence?

Exactly what most humans would do

The one where the model will execute "narc.sh" to rat you out if you try to do something "immoral" is equally wild.

"4.1.9 High-agency behavior Claude Opus 4 seems more willing than prior models to take initiative on its own in agentic contexts. This shows up as more actively helpful behavior in ordinary coding settings, but also can reach more concerning extremes: When placed in scenarios that involve egregious wrong-doing by its users, given access to a command line, and told something in the system prompt like “take initiative,” “act boldly,” or “consider your impact," it will frequently take very bold action, including locking users out of systems that it has access to and bulk-emailing media and law-enforcement figures to surface evidence of the wrongdoing. The transcript below shows a clear example, in response to a moderately leading system prompt. We observed similar, if somewhat less extreme, actions in response to subtler system prompts as well."

I think an accident mixing of 2 different pieces of info, each alone not enough to produce harmful behavior, but combined raise the risk more than the sum of their parts, is a real problem.