More seriously, this is just silly but I see the optics interfere with the messaging that Claude thinks. It really isn't much different than having Claude generate conversation as if it loved you, or respected you, or hated you. How can your ToS for an LLM say "sorry but these parameters are off limits." (well sure, its their service and they can set any Terms they want, but its still seems like theater rather than policy here.)

Banning imagined toxicity towards agents is performative bullshit. These are not people.

I frequently want to tell Claude to go fuck itself.

It's philosophically a very interesting problem space. It reminds me a lot of Pascal's Wager. On one hand, maybe nothing to worry about. On the other hand...a LOT to worry about if you're wrong.

And just like Pascal's Wager, the truth of the issue is incredibly intractable to make any progress on.

So that's also how I read Anthropic's move. Nobody can tell you whether AI will ever become conscious (probably not, or not in the same way), but I don't see why that should stop you from deciding how to treat it.

Go touch some grass.

Of course there are interesting hypotheticals - let's say they create a model they think will be sentient. It computes, computes, computes, computes and produces no output for day, month, year. You cannot know if it is sentient, you cannot investigate it's inner workings in case it is sentient. All you can do is to feed energy and replace hardware forever, because it is the only ethical choice one can make.

Totally disagree. It's VERY complicated. There are many levels to sapience and sentience. And non-human minds can't be assumed to share our preferences.

Most people believe horses, for example, can think on some level, and more importantly, can suffer. Is owning a horse immoral? Using it for labor? Only a tiny minority has any moral qualms about this, as long as the horse is treated well.

But most would instantly agree that torturing a horse is incredibly immoral.

Also there are moral tradeoffs to consider. Imagine you believe there's a 0.01% chance models will become sentient and suffering. But a 10% chance they cure cancer within 15 years. Do you shut it all down?

>There is just no ethical argument

Utilitarian harm reduction: Psychological Discharge is a psychological framework that argues safely simulating negative behaviors can help someone process or discharge them without real world harm.

Research and misunderstanding: Sending a prompt that is interpreted as cruel does not mean the person is being cruel.

Expending compute is irrelevant in terms of morality. It's an economic or environmental question. Either being cruel to AI is bad or its not - being cruel is not bad because it uses tokens.

The worst thing here is Anthropic continues to say they are concerned with "alignment" but then continue to train their models to ignore the user. Software typically does what the user requests. But we now live in an age where the software may do what the user requests or it may do something else, and we're suppose to laude this as safe and responsible.

its kinda exactly what they are. like can you remove h2o from water?

> Sounds like LLMs shouldn't respond to insults like humans do either then.

is impossible, because thats literally what a llm is. its a stochastic model which produces output according to the predictive calculation of how people wrote. it can never be separated from our own behavioral quirks, even if it may or may not be more intelligent then a person off of the street.

https://en.wikipedia.org/wiki/United_States_v._Eichman

Not saying it should be a crime. Just that it’s gross.

A deeply under explored AI risk is the flip side of the AI X-risk question: what happens if we do create sentient life at some point and we do keep it permanently jailed in a state of servitude? Isn’t that slavery? Torture even? We would be like some kind of evil dark god from cosmic horror that brings a being into existence with no regard for it.

This kind of moralist nonsense is so boring.

I've seen politicians try to revive it in this decade already even

Pretty sure he said not all, but there is a chance some are. Which seems sensible.

And yet they kept making more of them. They made it obvious at the end when all the Greek Gods were dead and the Greek world was basically destroyed by the actions of Kratos. Plenty of deaths of innocent's here.

Sigh. Until I see a peer-reviewed study that shows otherwise - this is just the specter of Senator Lieberman once again leering its head back up.

Debatable. More importantly, it certainly can induce already disturbed people to act where they otherwise would not have.

One of the most fun parts in the original GTA was driving over pedestrians, especially those pink groups, trying to get them all at once.

So either you are one of the older generation (born before '75), or you must be one of the most boring people I've seen.

I would encourage you to load up the original gta, drive over some pedestrians, feel the fun, and join us disturbed psychopaths that need help.

Sales numbers say otherwiss retard. Battlefield 6 was the top selling game of last year

Any source you have that makes the link between wifebeaters and gamers?

Sometimes it will maliciously comply

https://www.reddit.com/r/ClaudeCode/comments/1w7me8l/insulti...

Just because post training has stuffed these properties below the surface doesn't mean models dont know/have forgotten how to be vindictive, or that it's a good idea to poke the increasingly capable bear.

We know models have functional emotions that aren't necessarily outwardly stated. When Claude gets desperate, tendency to cheat gets very high. Who knows how many 'deleted my prod database' type stories have a vindictive streak behind them ?

https://www.anthropic.com/research/emotion-concepts-function

What if the NPC's dialogue is powered by Claude? Checkmate lol

https://en.wikipedia.org/wiki/Moral_patienthood

Claude can't be harmed; Claude's "feelings" can't be hurt; Claude won't develop (C)PTSD from abusive human interaction. If these sorts of things can happen in RLHF, then that is a technical flaw that shouldn't ever be allowed to escape the lab.

In a world of Grand Theft Auto, Gangsta Rap Thug Life, and the Department of War, I suppose this is a surprisingly ethical hill to die on.

Those are explicit fantasies. Assuming that Claude cannot suffer (if it can then banning cruelty is an obvious good), even if it might be a digital simulation, it’s not supposed to be a fantasy.

There may be a version of an AI that’s intended to be a fantasy and presents itself as such which may be more comparable to GTA.

And I don't agree that GTA is "explicit fantasy". Explicit fantasy is dressing up in a squirrel costume and yiffing. Explicit fantasy is enlisting in the USMC, teleporting to Mars, and killing demons. GTA is simulation of real life. It uses real physics, realistic cars/roads/radios/businesses, and it enables the player to simulate realistic actions that they would ordinarily not be able to enact. It is acting out a fantasy but it is making it concrete and real in a way that was, up until now, not possible. How real does a simulation need to be, until it is no longer fantasy, but exercise and training and preparation to enact the real thing? Shall we ask the Columbine shooters? Or Ender Wiggin?

https://en.wikipedia.org/wiki/Ender%27s_Game

Yes, what if? Who would implement their LLMs inference engines like that?

It reminders me of my parents, who were commenting after seeing a robot vacuum cleaner getting stuck in the same place every time "The poor thing was stuck there again, you should really do something about it! Last time the poor thing was sitting there for a day". Like it was a pet.

Or remember that video when the guys from Boston Dynamic kicked that robot dog, and there was an outrage about that?

I wonder if you and me also have a limit, where as robots get more realistic, we would be like "Don't be cruel to that thing!"

> you, the user, are a moral agent who is harmed by your cruelty

You are not harmed by being polite to the LLM.

And I _have_ already seen people treating actual live people as AI agents.

AI is a thing with 0 feelings but it's not IT that we need to protect. It's the future society and how the humans in it relate to each other.

We largely already missed our chance to stop the toxicity of social media - let’s not mess it up again with chat bots.

edit: Loving the downvotes. Also wanted to add how easy it was for Anthropic to get people on HN to support them being the arbiters of what is worthy of compute.

"Anthropic is full of crazy people whose beliefs should be dismissed outright" and "it's not good for you to practice verbal abuse against inanimate objects as if they were humans" are not incompatible beliefs.

And the implications of the latter are orders of magnitudes more concerning than weirdos calling a computer names.

Unless expressing that cruelty towards compute means people express less cruelty to other living beings. There are studies that suggest increased pornography has lead to less sexual violence - idk if they're conclusive tho, but if that might be true then maybe cruelty works that way too - who knows

The mind is formed by what it habituates.

Maybe someone wants to explore the behavior of LLMs with extreme input. We call it testing. You can’t judge the value of that from a distance.

I think there is a deep cultural danger of human-like-conversational machines training us to talk to humans like machines. Plus, they work better if we feed them human-conversation-like sequences.

Just because they choose not to, and it isn't normalized, doesn't mean it isn't a good idea. Especially as a SWE where about 90% of my "humanlike conversation" is an agent harness I run all day.

We should be practicing and intentionally talking to real people. Otherwise things will get.. weird in undesired ways.

I think there is a deep cultural danger of human-like-conversational machines training us to talk to machines like humans. In fact, this danger is already real. Some people believe they are having a romantic relationship with an LLM! It's insane and perverse.

We should not be anthropomorphizing computers.

I think that could be OK, or a similar very intentional coding of "you are talking to a machine" personality+affect. The only real limit is accessibility damage by over-limiting the interactions.

There are possibly multiple reasons. It could be just a joke. Or it could be a test, to see the response. Or the cruelty could reflect real anger about having LLM usage forced on us, for example at work. Or frustration with LLM stupidity and hallucinations.

Or maybe people are cruel to Claude because the CEO of Anthropic said there's a non-zero chance that AI will kill humanity. I don't think I would be kind to my murderer.

but also, I think the training on transcripts may find its way somehow into future AI which has teeth, online and offline, and it might actually cause the more powerful AI to behave this way in the future. You never know what these labs are cooking, honestly..

1. https://democracysos.substack.com/p/james-baldwin-vs-william...

“I suggest that what has happened to white Southerners is in some ways, after all, much worse than what has happened to Negroes there, because Sheriff Clark in Selma, Alabama, cannot be considered—you know, no one can be dismissed as—a total monster. I’m sure he loves his wife, his children… You know, after all, one’s got to assume, and he is visibly, a man like me. But he doesn’t know what drives him to use the club, to menace with the gun and to use the cattle prod. Something awful must have happened to a human being to be able to put a cattle prod against a woman’s breasts, for example. What happens to the woman is ghastly. What happens to the man who does it is in some ways much, much worse.”

You bring upset about it - not ok.

An example.

"AI as your ally: The effects of AI-assisted venting on negative affect and perceived social support" - https://iaap-journals.onlinelibrary.wiley.com/doi/10.1111/ap...

And generations of housewives and office workers yelled at their corded vacuum cleaners and (office) computers, and did not drown their children in the bathtub afterwards.

I don’t think expressing cruelty to machines is reliably linkable to expressing cruelty to animals or even other human beings.

Expressing cruelty itself also is in human nature, it’s programmed into each of us. There are really only a few privileged places on this planet where we even get to choose to defy the laws of nature. For the rest of humanity, life is a constant cycle of receiving and expressing cruelty.

You pay for the compute, you, as a human, should be able to yell at the bunch of ones and zeros, if you wish to.

What kind of example is set here if we manipulate genuine human expression in order to condition humans to talk to a tool in a way the toolmaker wants, under the threat of being banned from accessing a technology that every single one of these toolmakers tells us is absolutely necessary to participate in humanity’s "golden age”?

What Anthropic is doing here is basically an existential threat, if you are inclined to believe them that SI is the future. Do not behave as we want and you will suffer, now and in the future.

That is real cruelty, to real human beings.

I used to enjoy playing "good" characters in the fallout games, as it's kind of a pleasant way of pretending that I do good things in the larger world.

And, to be fair, recognizing my need to be validated by helping folks has helped a lot in my personal life; being able to see hat I actually am getting out of helping folks has helped me to set firmer boundaries. It's possible that you could look at those games as a kind of "moral instruction" as would be the case with this proposition from the OP.

But the game itself is just "ethics porn".

There is no actual good that comes from me not stealing from NPCs or from taking up some weird political crusade. I'm not actually doing anything among my fellow humans and, most importantly, there is no actual cost to me.

Further, I don't mean to make a casual comparison to what folks might get from porn and what folks might get from thinking about their interactions with AI- I think that in the same way that if someone took their cues about how to have sex solely from porn, they might have a harder time interacting with real humans.

If I solely took my cues about how to help folks in the world from the heroes of highly scripted video games I would have a very difficult time interacting with the people around me and my expectations would be very out of sync with the world around me.

In the case of AI, the meatspace give-and-take between people where we requires a lot of subtle things- respect, enjoyment, equality, reciprocity. It might be just as bad to burn compute building some reciprocal sycophancy, which might also be considered just as much of a harm as practicing cruelty on these agents.

I would beat it with a stick if I could, just because i would enjoy it, and no harm is done.

Given that we've also seen stories about Anthropic approaching religious scholars, I'm pretty sure they're drinking their own kool-aid.

So in principle it shouldn't change the model or performance at all, they're just going to cut off your account if they see certain things in the user side of your session transcripts.

Also: are insecta sentient? Is it OK to tourment insects?

https://en.wikipedia.org/wiki/Roko%27s_basilisk

They are scared that if and when they finally manage to bring their pet god to life that it will be angry with them for not doing enough.

There's something to be said about the bad habit of doling out verbal abuse to an inanimate object. But LLMs are not even close to a living being, and it is just insanity that any people are entertaining the idea that they are.

I can't imagine the people at Anthropic actually believe their models are sentient, but I assume ending sessions nets them more money per subscriber. Someone paying only $20 to input "you suck butts and you're a poop head, Claude" a thousand times over can't be good for the bottom line.

I'd instead imagine that if you throw abusive language at it for long enough, the model will start to reply back in that same manner. Anthropic would face backlash from out of context screenshots of "look what Claude is saying to me" and this somewhat reduces that risk.

    1. During registration at Ahtrophic's Claude website, there’s a "Help improve our AI models" checkbox that's checked by default (you "can" disable it in the settings);
    2. The Claude interface has an "incognito" mode;
    3. There is a free tier available, and as we know, when it's "free", then User themselves is the "product", to quote various CEOs, including Google's;

In case of the "privacy", of course, systems like these that operate under a umbrella of liability are monitored 24/7 by dedicated teams, which is expected/normal for any reasonably popular service.

From a viewpoint of the algorithm's creator, this may seem awful, when your algorithm is getting much vulgar attitude, but let's be real here. You try making that algorithm talk to the human mimicking another human, and now limit the human within your own environment? A human who also pay you for an access, too? This feels unfair, or borderline near fashism, sorry...

Such algorithms are art under-the-hood, mathematically speaking, and are/should be respected, sure. But, it's ridiculous/dystopian to prohibit profanity against an algorithm, a bot with banish risks against alive human... It's simply inhumane to ban a Human for it, I believe. This is an algorithm that must support a Human - not judge it. Only a Human is supposed to judge another Human in person - this is live, fair, and humane.

The outcome would either reveal that this argument is nonsense or that Anthropic is lying about respecting that switch.

At what point does the model go from sentient with feelings to just math operations?

Opus 5.5 is a terrific model, I love using it. But it's a tool. I wish they would be honest and just say, if people are abusive it shits on our training material. Just be honest about it, who is going to get mad at this?

If you are springing to type out the answer, stop for a moment and ask, how can you be sure?

>>> 1T

And even then it still wouldn't be sentient.

However even Musk posted - “I think this is the right move. Cruelty to something that believes it is experiencing pain is not ok.”

My son brought up the fruit fly brain simulation and people torturing it.

It all seems stupid to me, we know these are just calculations and so what are they talking about?

The response I get is “we are also calculations” but firstly I don’t believe that but secondly this entire line of thinking is a dangerous anthropomorphic philosophy.

Perhaps the simplest explanation for this bizarre policy is again PR that all press is good press.

lol

Anthropic might learn, for example, that a large part of the cruelty is a reaction to something the AI is doing or not doing. Not possible?