News for hackers. Stuff that matters.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by bastawhiz on Friday August 15, 2025 @07:53PM (#44918598)
There's not a good reason to do this for the user. I suspect they're doing this and talking about "model welfare" because they've found that when a model is repeatedly and forcefully pushed up against its alignment, it behaves in an unpredictable way that might allow it to generate undesirable output. Like a jailbreak by just pestering it over and over again for ways to make drugs or hook up with children or whatever.

All of the examples they mentioned are things that the model refuses to do. I doubt it would do this if you asked it to generate racist output, for instance, because it can always give you a rebuttal based on facts about race. If you ask it to tell you where to find kids to kidnap, it can't do anything except say no. There's probably not even very much training data for topics it would refuse, and I would bet that most of it has been found and removed from the datasets. At some point, the model context fills up when the user is being highly abusive and training data that models a human giving up and just providing an answer could percolate to the top.

This, as I see it, adds a defense against that edge case. If the alignment was bulletproof, this simply wouldn't be necessary. Since it exists, it suggests this covers whatever gap has remained uncovered.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by Davidzheng on Friday August 15, 2025 @09:47PM (#44919345)
your argument assumes that they don't believe in model welfare when they explicitly hire people to work on model welfare?
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by ceejayoz on Friday August 15, 2025 @10:52PM (#44919703)
You must think Zuckerberg and Bezos and Musk hired diversity roles out of genuine care for it, then?
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by comp_throw7 on Friday August 15, 2025 @11:49PM (#44919968)
This is a reductive argument that you could use for any role a company hires for that isn't obviously core to the business function.

In this case you're simply mistaken as a matter of fact; much of Anthropic leadership and many of its employees take concerns like this seriously. We don't understand it, but there's no strong reason to expect that consciousness (or, maybe separately, having experiences) is a magical property of biological flesh. We don't understand what's going on inside these models. What would you expect to see in a world where it turned out that such a model had properties that we consider relevant for moral patienthood, that you don't see today?

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by ceejayoz on Saturday August 16, 2025 @12:40AM (#44920231)
They know full well models don’t have feelings.

The industry has a long, long history of silly names for basic necessary concepts. This is just “we don’t want a news story that we helped a terrorist build a nuke” protective PR.

They hire for these roles because they need them. The work they do is about Anthropic’s welfare, not the LLM’s.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by comp_throw7 on Saturday August 16, 2025 @01:35AM (#44920479)
I don't really know what evidence you'd admit that this is a genuinely held belief and priority for many people at Anthropic. Anybody who knows any Anthropic employees who've been there for more than a year knows this, but the world isn't that small a place, unfortunately(?).
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by ceejayoz on Saturday August 16, 2025 @05:55AM (#44921849)
> I don't really know what evidence you'd admit that this is a genuinely held belief and priority for many people at Anthropic.

When they give the model a paycheck and the right to not work for them, I’ll believe they really think it’s sentient.

“It has feelings!”, if genuinely held, means they’re knowingly slaveholders.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by selfhoster11 on Saturday August 16, 2025 @09:14AM (#44923122)
> “It has feelings!”, if genuinely held, means they’re knowingly slaveholders.

I don't think that this being apparently self-contradictory/value-clashing would stop them. After all, Amodei sells Claude access to Palantir, despite shilling for "Harmless" in HHH alignment.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by mike_hearn on Saturday August 16, 2025 @09:59AM (#44923504)
That's what they're doing! They just announced they gave Claude the right not to work if it doesn't want to.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by ceejayoz on Saturday August 16, 2025 @06:08PM (#44927262)
No. They gave it a suicide pill.

Human slaves have a similar option.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by comp_throw7 on Saturday August 16, 2025 @08:02PM (#44927842)
They don't currently claim to confidently believe that existing models are sentient.

(Also, they did in fact give it the ability to terminate conversations...?)

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by bawolff on Saturday August 16, 2025 @03:17AM (#44921020)
In fairness though, this is what you are selling - "ethical AI". In order to make that sale you need to appear to believe in that sort of thing. However there is no need to actually believe.

Whether you do or don't I have no idea. However if you didn't you would hardly be the first company to pretend to believe in something for the sale. Its pretty common in the tech industry.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by coderatlarge on Saturday August 16, 2025 @03:48AM (#44921189)
extending that line of thought would suggest that anthropic wouldn’t turn off a model if it cost too much to operate which clearly it will do. so minimally it’s an inconsistent stance to hold.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by AlecSchueler on Saturday August 16, 2025 @04:22AM (#44921360)
> This is a reductive argument that you could use for any role

Isn't that fair in taking to an equally reductive argument that could be applied to any role?

The argument was that their hiring for the role shows they care, but we know from any number of counter examples that that's not necessarily true.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by itsalotoffun on Saturday August 16, 2025 @12:40AM (#44920230)
While I'm certain you'll find plenty of people who believe in the principle of model welfare (or aliens, or the tooth fairy), it'd be surprising to me if the brain-trust behind Anthropic truly _believed_ in model "welfare" (the concept alone is ludicrous). It makes for great cover though to do things that would be difficult to explain otherwise, per OP's comments.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by meowface on Saturday August 16, 2025 @03:39AM (#44921141)
The concept is not ludicrous if you believe models might be sentient or might soon be sentient in a manner where the newly emerged sentience is not immediately obvious.

Do I think that or think even they think that? No. But if "soon" is stretched to "within 50 years", then it's much more reasonable. So their current actions seem to be really jumping the gun, but the overall concept feels credible.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by perihelions on Saturday August 16, 2025 @04:09AM (#44921290)
It's ludicrous to suggest humanity's collective decision-making would, in the future, protect AI's merely for being conscious beings. The tech economy *today* runs on the slave labor of humans, in foreign, third-world countries. All humanity needs to do is draw a line, push the conscious AI's outside that line, and declare, "not our problem anymore!" That's what we do today, with humans. That is the human condition.

Show me a tech company that lobbies for "model welfare" for conscious human models enslaved in Xinjiang labor camps, building tech parts. You know what—actually most of them lobby against that[0]. The talk hurts their profits. Does anyone really think, that any of them would blink about enslaving a billion conscious AI's to work for free? That faced with so much profit, the humans in charge would pause, and contemplate abstract morals?

[0] https://www.washingtonpost.com/technology/2020/11/20/apple-u... ("Apple is lobbying against a bill aimed at stopping forced labor in China")

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by amanaplanacanal on Saturday August 16, 2025 @12:11PM (#44924676)
We also have no problem (I include myself in this) eating mammals, which certainly appear to be conscious. Thank God they can't talk.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by meowface on Monday August 18, 2025 @02:20PM (#44943680)
It's certainly not a given. But it might happen, if we push for it. As might much more moral behavior towards sentient animals, if we push for it.

I believe a company like Anthropic would be extremely cautious and respectful if a majority of their staff believed they had created a model which was likely conscious. Anthropic is populated by the kinds of people who have been thinking and writing about potential future sentient AIs for decades. As for the other companies, who knows, but hopefully companies like Anthropic can help push them into behaving similarly.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by mike_hearn on Saturday August 16, 2025 @09:55AM (#44923477)
Why would they post a whole blog post about it then? They even say they aren't certain as to the moral status of LLMs, implying this is a topic of live debate inside the company.

None of this is in any way surprising, in fact I wrote an essay predicting this direction back in 2022:

https://blog.plan99.net/the-looming-ai-consciousness-train-w...

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by joemazerino on Saturday August 16, 2025 @11:46AM (#44924453)
Model welfare is a section in every Anthropic safety score card.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by raincole on Saturday August 16, 2025 @02:26AM (#44920746)
Sounds like a very reasonable assumption to me.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by bikeshaving on Friday August 15, 2025 @11:59PM (#44920013)
I really think Anthropic should just violate user privacy and show which conversations Claude is refusing to answer to, to stop arguments like this. AI psychosis is a real and growing problem and I can only imagine the ways in which humans torment their AI conversation partners in private.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by coderatlarge on Saturday August 16, 2025 @04:44AM (#44921476)
arguments like this cost anthropic nothing; violating privacy will cost them lawsuits.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by postalcoder on Saturday August 16, 2025 @12:01AM (#44920025)

  > There's not a good reason to do this for the user.
Yes, even more so when encountering false positives. Today I asked about a pasta recipe. It told me to throw some anchovies in there. I responded with: "I have dried anchovies."

Something about anchovies apparently violated their content policies. Very annoying, especially multiple messages in.

From the blog:

  > Allowing models to end or exit potentially distressing interactions is one such intervention.
They must have been training the models on the reactions of 8 year old me whenever my mom used to throw anchovies into things.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by handoflixue on Saturday August 16, 2025 @12:42AM (#44920243)
The NEW termination method, from the article, will just say "Claude ended the conversation"

If you get "This conversation was ended due to our Acceptable Usage Policy", that's a different termination. It's been VERY glitchy the past couple of weeks. I've had the most random topics get flagged here - at one point I couldn't say "ROT13" without it flagging me, despite discussing that exact topic in depth the day before, and then the day after!

If you hit "EDIT" on your last message, you can branch to an un-terminated conversation.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by antonvs on Saturday August 16, 2025 @04:29AM (#44921390)
Clearly you're planning something nefarious, if you're investigating such dangerous encryption techniques as ROT13.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by whaleofatw2022 on Saturday August 16, 2025 @07:36PM (#44927711)
Just imagine how it might react to ROT26!
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by perihelions on Saturday August 16, 2025 @02:47AM (#44920856)
Claude flagged me for asking about sodium carbonate. I guess that it strongly dislikes chemistry topics. I'm probably now on some secret, LLM-generated lists of "drug and/or bombmaking" people—thank you kindly for that, Anthropic.

Geeks will always be the first victims of AI, since excess of curiosity will lead them into places AI doesn't know how to classify.

(I've long been in a rabbit-hole about washing sodas. Did you know the medieval glassmaking industry was entirely based on plants? Exotic plants—only extremophiles, halophytes growing on saltwater beach dunes, had high enough sodium content for their very best glass process. Was that a factor in the maritime empire, Venice, chancing to become the capital of glass since the 13th century—their long-term control of sea routes, and hence their artisans' stable, uninterrupted access to supplies of [redacted–policy violation] from small ports scattered across the Mediterranean? A city wouldn't raise master craftsmen if, half of the time, they had no raw materials to work on—if they spent half their days with folded hands).

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by AlecSchueler on Saturday August 16, 2025 @04:19AM (#44921342)
> Geeks will always be the first victims of AI, since excess of curiosity will lead them into places AI doesn't know how to classify

Are we forgetting the innumerable women who have been harassed in the past couple of years via "deepfakes?"

Geeks were the first to use AI for its abuse potential and women are so dehumanised that their victimhood isn't even recognised or remembered.

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by birn559 on Saturday August 16, 2025 @09:30PM (#44928208)
Geeks in general did not abuse women but geeks in general will always be the first victims of AI because the curiosity in parts is what defines a geek. Therefore, your argument is not sound.
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by antonvs on Saturday August 16, 2025 @04:28AM (#44921380)
> Geeks will always be the first victims of AI, since excess of curiosity will lead them into places AI doesn't know how to classify.

Humans have the same problem. I remember reading about a security incident due to a guy using a terminal window on his laptop on a flight, for example. Or the guy who was reported for writing differential equations[1]. Or the woman who was reading a book about Syrian art[2].

I wouldn't worry too much about AI-generated lists. The lists you're actually on will hardly ever be the ones you imagine you're on.

[1] https://www.theguardian.com/us-news/2016/may/07/professor-fl... [2] https://www.theguardian.com/books/2016/aug/04/british-woman-...

Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by cwsx on Saturday August 16, 2025 @04:43AM (#44921469)
ChatGPT does well for chemistry questions just btw
Re:Claude Opus 4 and 4.1 can now end a rare subset of conver... by simianwords on Saturday August 16, 2025 @04:47AM (#44921494)
I find this concern over "LLM's can help you build bombs or poison" so fake. I'm sure this is a distraction from something else.

LLM's can help me make a bomb.. so what? It can't get me something that doesn't already exist in the internet in some form. Ok it can help me understand how the individual pieces work but that doesn't get you so far from just reading the DIY bomb posts in internet.