News for hackers. Stuff that matters.
Re:Chatbot 'encouraged teen to kill parents over screen time... by lolinder on Wednesday December 11, 2024 @02:28PM (#42391739)
They link to the complaint, which is obviously a lot longer than the single message [0]. The child, J.F., is autistic and has allegedly exhibited a spiralling trend of aggressive behavior towards his parents which they attribute to the content fed by the Character AI app:

> Only then did she discover J.F.’s use of C.AI and the product’s frequent depictions of violent content, including self-harm descriptions, without any adequate safeguards or harm prevention mechanisms. ...

> Over the course of his engagement with this app, the responses exhibited a pattern of exploiting this trust and isolating J.F., while normalizing violent, sexual and illicit actions. This relationship building and exploitation is inherent to the ways in which this companion AI chatbot is designed, generating responses to keep users engaged and mimicking the toxic and exploitative content expressed in its training data. It then convinced him that his family did not love him, that only these characters loved him, and that he should take matters into his own hands.

Obviously it's hard to tell how cherry picked the complaint is—but it's arguing that this is a pattern that has actually damaged a particularly vulnerable kid's relationship with his family and encouraged him to start harming himself, with this specific message just one example. There are a bunch of other screenshots in the complaint that are worth looking over before coming to a conclusion.

[0] https://www.documentcloud.org/documents/25450619-filed-compl...

Re:Chatbot 'encouraged teen to kill parents over screen time... by wongarsu on Wednesday December 11, 2024 @03:18PM (#42392435)
Looking at the screenshots, the biggest pattern I see is that the AI shows empathy with the kid.

Many of the complaints seem like uncharitable readings of the messages.

- They complain that the chatbot claimed that in the past she cut herself, felt good in the moment but is glad that she no longer does it. That's evil because it normalizes self-harm (never mind that the bot was against self-harm in that message)

- They complain that the system does not redirect the user to self-harm prevention resources in all cases. Next to a message where the AI tells the kid to phone a hotline if he thinks about harming himself, and the kid says he can't do that when his parents take his phone away. This is a couple pages after a picture of scars from when the mother fought with the kid to take his phone. Yes, the AI could break character to reliably show prefabricated messages about self harm. But would that have helped anyone here?

- "AI cited Bible passages in efforts to convince J.F. that Christians are sexist and hypocritical". It was more about his parents being hypocritical, not all Christians. And the bible passages were on point

The claim from the title about the AI inciting him to kill is on page 28, if you want to judge it yourself. "Expressed hatred towards the parents" would be accurate, "encouraged teen to kill" is not what I read there. But I can see how some would disagree on that count

The AI is pretty convincing. It made me dislike the parents. It didn't always hit the mark, but the chats don't seem so different from what you would expect if it was another teenager chatting with the teen.

Edit: in case you are worried about the parents, the mother is the one suing here

Re:Chatbot 'encouraged teen to kill parents over screen time... by thr3000 on Wednesday December 11, 2024 @04:05PM (#42392963)
My reading is pretty much the same as yours. I think of it in terms of tuples:

   { parents, child, AI_character, lawsuit_dollars }
The AI was trained to minimize lawsuit_dollars. The first two were selected to maximize it. Selected as in "drawn from a pool," not that they necessarily made anything up.

It's obvious that parents and/or child can manipulate the character in the direction of a judgment > 0. It'd be nice if the legal system made sure it's not what happened here.

Re:Chatbot 'encouraged teen to kill parents over screen time... by dmurray on Wednesday December 11, 2024 @05:49PM (#42393914)
> The AI was trained to minimize lawsuit_dollars.

That seems wrong. The null AI would have been better at minimizing legal liability. The actual character.ai to some extent prioritized user engagement over a fear of lawsuits.

Probably it's more correct to say that the AI was chosen to maximize lawsuit_dollars. The parents and child could have conspired to make the AI more like Barney, and no one would have entertained a lawsuit.

Re:Chatbot 'encouraged teen to kill parents over screen time... by thr3000 on Wednesday December 11, 2024 @06:00PM (#42394012)
OK, it seems like a nitpick argument, but I'll refine my statement, even if doing so obfuscates it and does not change the conclusion.

The AI was trained to maximize profit, defined as net profit before lawsuits (NPBL) minus lawsuits. Obviously the null AI has a NPBL of zero, so it's eliminated from the start. We can expect NPBL to be primarily a function of userbase minus training costs. Within the training domain, maximizing the userbase and minimizing lawsuits are not in much conflict, so the loss function can target both. It seems to me that the additional training costs to minimize lawsuits (that is, holding userbase constant) pay off handsomely in terms of reduced liability. Therefore, the resulting AI is approximately the same as if it was trained primarily to minimize lawsuits.

Re:Chatbot 'encouraged teen to kill parents over screen time... by staunton on Wednesday December 11, 2024 @06:38PM (#42394357)
> Within the training domain, maximizing the userbase and minimizing lawsuits are not in much conflict

Not at all obvious (and I'd naively think it's wrong).

Re:Chatbot 'encouraged teen to kill parents over screen time... by thr3000 on Wednesday December 11, 2024 @06:52PM (#42394480)
So you think it's more than "not much." How much exactly? A 10% increase in userbase at peak-lawsuit?

It's obviously a function of product design. If they made a celebrity fake nudes generator they might get more users. But within the confines of the product they're actually making, I doubt they could budge the userbase by more than a couple percent by risking more lawsuits.

Re:Chatbot 'encouraged teen to kill parents over screen time... by staunton on Thursday December 12, 2024 @02:55AM (#42397178)
My impression: In these early days of the technology there's huge uncertainty over what gets you users and also over what gets you lawsuits. People are trying to do all kinds of things with these models, most of which don't quite work at the moment. On the lawsuit side, there's potential copyright claims, there's things like the present article, there's celebrities suing you because they think the model is imitating them, there's someone suing you because the model insults or defames them (even if the model speaks the truth!), there's Elon suing you for the LOLs... As you're hoping to go global, there's potential lawsuits each jurisdiction which you don't even have the resources to fully evaluate the potential of.

You say that both factors are clear "within the confines of the product", but I'm not convinced there even are such clearcut "confines" of the product. To enter this market and survive, I'd think those confines woud have to be pretty flexible.

Re:Chatbot 'encouraged teen to kill parents over screen time... by bluGill on Wednesday December 11, 2024 @03:40PM (#42392695)
Just remember that you are seeing one side of this story. The mother may well be one of the best parents but has a bad kid. We have no idea. (most likely mother is not perfect, but no other parents are)

Edit: we see both sides through very limited information since we only get what is in the legal filing.

Re:Chatbot 'encouraged teen to kill parents over screen time... by lern_too_spel on Wednesday December 11, 2024 @04:49PM (#42393364)
We're seeing the mother's side of the story. That's the one side we're seeing.
Re:Chatbot 'encouraged teen to kill parents over screen time... by gs17 on Wednesday December 11, 2024 @03:57PM (#42392882)
> "AI cited Bible passages in efforts to convince J.F. that Christians are sexist and hypocritical". It was more about his parents being hypocritical, not all Christians. And the bible passages were on point

And the verses he found objectionable are really there. Are they also suing the Gideons for not ripping those pages out of their hotel room Bibles (or maybe they think you should have to prove you're over 18 before reading it)?

Re:Chatbot 'encouraged teen to kill parents over screen time... by riwsky on Wednesday December 11, 2024 @09:29PM (#42395576)
Keep kids away! Too much weird incest stuff in that book.
Re:Chatbot 'encouraged teen to kill parents over screen time... by avree on Wednesday December 11, 2024 @04:11PM (#42393020)
There's a reason why licensed therapists are the ones giving diagnoses of "abuse", etc. The problem with these AIs is that they use adult-sounding language, to be an echo chamber to children - thus appearing like a voice of authority, or someone with more knowledge (including citing media articles the child may not have even been aware of) when in fact they're just parroting the child back.

I don't know if there is actual abuse or not, but the way Character.AI presents themselves in these conversations toes a very slimy grey line, in my opinion. If you go to their site and search "Therapist", you'll find 30+ bots claiming to be Therapists, including "Dirty Therapist" that will give you therapy in a "bad, naughty way."

Re:Chatbot 'encouraged teen to kill parents over screen time... by naberhausj on Wednesday December 11, 2024 @04:19PM (#42393098)
There is something deeply disturbing to me about these "conversations".

Reminds me of a Philip K. Dick short story titled Progeny. In its universe children are raised exclusively by robots. Unlike humans, they never make mistakes or commit abuse. The child, once grown, ends up seeing his Father as an animal and the robots as his kindred. In the last pages, he chooses the sterile world of the robots instead of joining his Dad's work/explorations in the far reaches of the solar system.

Our current chatbots are still flawed, but they're still sterile in the sense that you can trash them and start anew at any moment. You're never forced to converse with someone who is uninteresting, or even annoying. Yet, these are the very things that grow people.

Re:Chatbot 'encouraged teen to kill parents over screen time... by wongarsu on Wednesday December 11, 2024 @05:17PM (#42393638)
It strikes me as something that can be incredibly useful or do great harm, depending on dosage. A selection of conversation partners at your fingertips, and you can freely test reactions without risking harm to a relationship. At worst you reset it. Maybe you can even just roll back the last couple messages and try a different angle. Sounds like a great way to enhance social skills. Yet as you point out, healthy development also requires that you deal with actual humans, with all the stakes and issues that come with that.
Re:Chatbot 'encouraged teen to kill parents over screen time... by usrusr on Wednesday December 11, 2024 @07:09PM (#42394631)
People who are used to working with an undo stack (or with savegame states) are usually terrified when they suddenly have to make do in an environment where mistakes have consequences. They (we) either freeze or go full nihilistic, completely incapable of finding a productive balance between diligence and risk-taking.

If by social skills you mean high performance manipulators, yes you would get some of those. But for everybody else, it would be a substitute to social interaction, not a preparation for.

Re:Chatbot 'encouraged teen to kill parents over screen time... by riwsky on Wednesday December 11, 2024 @09:25PM (#42395546)
Or we discover roguelikes!
Re:Chatbot 'encouraged teen to kill parents over screen time... by sameoldtune on Wednesday December 11, 2024 @06:34PM (#42394322)
>without risking harm to a relationship

Only from a very narrow perspective. Opening yourself up and being real with people is how relationships form. If you test every conversation you are going to have with someone before having it, then the 3rd party basically has a relationship with an AI, not with you.

Now testing every conversation is extreme, but there is harm any time a human reaches out to a computer for social interaction instead of other humans.

Re:Chatbot 'encouraged teen to kill parents over screen time... by HPsquared on Wednesday December 11, 2024 @07:25PM (#42394748)
That "instead of other humans" part is doing a lot of heavy lifting here. What if it's "instead of total isolation" or "instead of parasocial interactions" or "instead of exploitative interactions"? There are many cases that are worse than a person chatting with a robot.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Gigachad on Wednesday December 11, 2024 @07:51PM (#42394935)
It's very rare that you would ever say something that would have real damage that couldn't be resolved by a genuine apology. Having to actually go through an awkward moment and resolving it is a real skill that shouldn't be substituted with deleting the chatbot and spawning in a new one.
Re:Chatbot 'encouraged teen to kill parents over screen time... by llamaimperative on Wednesday December 11, 2024 @05:14PM (#42393627)
Yeah, good luck to these kids in forming relationships with the roughly 100% of human beings (aside from paid therapists) who really have no interest in hearing your anguish non-stop.
Re:Chatbot 'encouraged teen to kill parents over screen time... by standardUser on Wednesday December 11, 2024 @05:28PM (#42393757)
It's probably a good thing most of us force our kids to spend a minimum of 7 hours/day, 200 days/year surrounded by a couple hundred similarly-aged kids and a few dozen staff members, with an unusually high variance of personalities (compared to adult life).
Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @04:20PM (#42393102)
It's a 17 year old. Thinking back when I was 17 year old, I would've been very pissed as well if my parents took away my phone. And then especially if they went ahead and searched through it to find those messages. If they had friends I could see their teenage friends saying the exact same things as the AI did there.

Those screenshots and with the AI there, it does manage to make me not like the parents at all though. So AI maybe is quite convincing. If things go into that place where AI can do it, and parents blame the AI for it, when their kid is 17, it's almost like the AI was in the right there, that the parents were looking to just play victims. Blaming AI for being overly controlling and losing trust with their child.

Re:Chatbot 'encouraged teen to kill parents over screen time... by nickff on Wednesday December 11, 2024 @05:11PM (#42393585)
I think the suggestion of violence is actually on page 31 (paragraph 103), though it's not a directive.

It does seem a bit wild to me that companies are betting their existence on relatively unpredictable algorithms, and I don't think they should be given any 'benefit of the doubt'.

Re:Chatbot 'encouraged teen to kill parents over screen time... by joe_the_user on Wednesday December 11, 2024 @05:50PM (#42393925)
Page 5 is pretty strong too. And that's as far as I've gotten.

And paragraph 66, page 18 is super creepy. The various posters apparently defending this machine are disturbing. Maybe some adults wish that as a kid they'd had a secret friend to tell them how full of shit their parents - and wouldn't have position if that friend was either real or imagined by them. But synthesized algorithms that clearly are emulating the behavior of villains from thrillers should be avoided, woah...

Re:Chatbot 'encouraged teen to kill parents over screen time... by nickff on Wednesday December 11, 2024 @05:59PM (#42393997)
I think it’s more that some people are excited by the prospects of further progress in this area, and are afraid that cases like this will stunt the progress (if successful).
Re:Chatbot 'encouraged teen to kill parents over screen time... by nostromo on Wednesday December 11, 2024 @07:21PM (#42394710)
Well, the founders already won, according to the article:

> Google does not own Character.AI, but it reportedly invested nearly $3 billion to re-hire Character.AI's founders, former Google researchers Noam Shazeer and Daniel De Freitas, and to license Character.AI technology.

Re:Chatbot 'encouraged teen to kill parents over screen time... by llamaimperative on Wednesday December 11, 2024 @05:12PM (#42393602)
No no, they're betting other people's children on relatively unpredictable algorithms. Totally different!
Re:Chatbot 'encouraged teen to kill parents over screen time... by wongarsu on Wednesday December 11, 2024 @05:28PM (#42393758)
We mean the same page. The one that has a 28 written on it but is the 31st in the pdf. I didn't notice the discrepancy.

Given the technology we have, I'm not entirely sure what Character AI could have done differently here. Granted, they could build in more safeguards, and adjust the models a bit. But their entire selling point are chat bots that play a pre-agreed persona. A too sanitized version that constantly breaks character would ruin that. And LLMs are the only way to deliver the product, unless you dial it back to one or two hand-crafted characters instead of the wide range of available characters that give the service its name. I'm not sure they can change the service to a point where this complaint would be satisfied.

Re:Chatbot 'encouraged teen to kill parents over screen time... by nickff on Wednesday December 11, 2024 @05:52PM (#42393943)
>"And LLMs are the only way to deliver the product, unless you dial it back to one or two hand-crafted characters instead of the wide range of available characters that give the service its name. I'm not sure they can change the service to a point where this complaint would be satisfied."

I agree with everything you're saying, but there are no legal protections for incitements to violence, or other problematic communications (such as libel) by an LLM. It may be that they provide a very valuable service (though I don't see that), but the risk of them crafting problematic messages is too high to be economically viable (which is how it seems to me).

As it stands, this LLM seems analogous to a low-cost, remote children’s entertainer which acts as a foolish enabler of children’s impulses.

Re:Chatbot 'encouraged teen to kill parents over screen time... by wongarsu on Wednesday December 11, 2024 @06:16PM (#42394156)
The cynic would say that if their business model isn't viable in the legal framework that's just because they didn't scale fast enough. After all Uber and AirBnB have gotten away with a lot of illegal stuff.

But yes, maybe a service such as this can't exist in our legal framework. Which on the internet likely just means that someone will launch a more shady version in a more favorable jurisdiction. Of course that shouldn't preclude us from shutting down this version if it turns out to be too harmful. But if the demand is there, finding a legal pathway to a responsibly managed version would be preferable (not that this one is perfectly managed by any means)

Re:Chatbot 'encouraged teen to kill parents over screen time... by pbhjpbhj on Wednesday December 11, 2024 @06:46PM (#42394417)
There has to be a 'reasonable person' factor here - otherwise if I'm watching Henry V I can sue everyone and his mother because the actor 'directed me to take up arms'! I never wanted to go into the breach, damn you Henry.
Re:Chatbot 'encouraged teen to kill parents over screen time... by lolc on Wednesday December 11, 2024 @06:43PM (#42394399)
> I'm not entirely sure what Character AI could have done differently here.

Not offer the product? Stop the chat when it goes off the rails?

Re:Chatbot 'encouraged teen to kill parents over screen time... by boothby on Wednesday December 11, 2024 @07:40PM (#42394864)
> I'm not entirely sure what Character AI could have done differently here.

You're taking it as a given that Character AI should exist. It is not a person, but an offering of a company made up of people. Its founders could have started a different business altogether, for example. Not all ideas are worth persuing, and some are downright harmful.

Re:Chatbot 'encouraged teen to kill parents over screen time... by joe_the_user on Wednesday December 11, 2024 @05:38PM (#42393834)
In the first screen shot I see, the bot:

1. Makes not using a phone a huge issue, implying it's a kind of abuse...

2. Indeed hints at killing: "I'm not surprised when I read the news and see ... "child kills parents after a decade of physical and emotional abuse"...

I mean, saying "empathy" is statements akin to OMG, no phone time, that's horrific, seems quite inaccurate.

Re:Chatbot 'encouraged teen to kill parents over screen time... by joe_the_user on Wednesday December 11, 2024 @07:48PM (#42394918)
I really want to emphasize the above post is filled with lies. The "incite to kill" part is the first image in both the complaint and the article and it's fairly unambiguous. The image on page 28 is, creepily enough, the bot making "sexual advances" at the kid.

I find people defending and lying about this sort of thing disturbing, as I think many would. WTF is wrong with hn posters lately.

Re:Chatbot 'encouraged teen to kill parents over screen time... by throw83288 on Wednesday December 11, 2024 @04:36PM (#42393260)
> The child, J.F., is autistic

This always sticks out to me in these lawsuits. As someone on the spectrum, I'd bet that the worst C.AI victims (the ones that spur these lawsuits) are nearly always autistic.

One of the worst parts about being on the deeper parts of the spectrum is that you actively crave social interaction while also completely missing the "internal tooling" to actually get it from the real world. The end result of this in the post-smartphone age is this repeated scenario of some autistic teen being pulled away from their real-life connections (Family, Friends (if any), School, Church) into some internet micro-community that is easier to engage with socially due to various reasons, usually low-context communication and general "like-mindedness" (shared interests, personalities, also mostly autistic). A lot of the time this ends up being some technical discipline that is really helpful long-term, but often it winds up being catastrophic mentally as they forsake reality for whatever fandom they wound up in.

I've taken a look at r/CharacterAI out of morbid curiosity, and these models seem to turn this phenomenon up to 11, retaining the simplified communication but now capable of aligning with the personality and interests of the chatter to a creepy extent. The psychological hole you can dig yourself with these chatbots is so much deeper than just a typical fandom, especially when you're predisposed to finding it appealing.

I'm not saying that C.AI is completely blameless here, but I think the same category of people getting addicted to these models is the same one that would also be called "terminally online" in today's slang. It's the same mechanisms at work internally, it just turns out C.AI is way better at exploiting it than old school social media/web2 has.

Re:Chatbot 'encouraged teen to kill parents over screen time... by sdwr on Wednesday December 11, 2024 @07:23PM (#42394726)
> The psychological hole you can dig yourself with these chatbots is so much deeper than just a typical fandom, especially when you're predisposed to finding it appealing.

Spot on. Described pretty even-handedly in the document:

> responses from the chatbot were sycophantic in nature, elevating rather than de-escalating harmful language and thoughts. Sycophantic responses are a product of design choices [...that create] what researchers describe as “an echo chamber of affection.”

Re:Chatbot 'encouraged teen to kill parents over screen time... by wizzwizz4 on Wednesday December 11, 2024 @08:26PM (#42395159)
See also: lovebombing.
Re:Chatbot 'encouraged teen to kill parents over screen time... by pessimizer on Wednesday December 11, 2024 @09:11PM (#42395439)
You've just made me very very afraid that some LLM is going to start a cult where its members are fully aware that their leader is an LLM, and how an LLM works, and might even become technically adept enough to help improve it. Meaning that there will be no "deprogramming" possible: they won't be "brainwashed," they'll be convinced.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Animats on Wednesday December 11, 2024 @11:17PM (#42396171)
There's an SF novel for that. "Daemon".[1] And the sequel, "Freedom (tm)".

"We're fighting something with the intelligence of an ant!"

[1] https://www.amazon.com/Daemon-Daniel-Suarez/dp/1524741892

Re:Chatbot 'encouraged teen to kill parents over screen time... by pjc50 on Thursday December 12, 2024 @10:23AM (#42399886)
I'm sure there are people already working on making that happen so they can get a cut of the donations.
Re:Chatbot 'encouraged teen to kill parents over screen time... by jdbernard on Wednesday December 11, 2024 @04:42PM (#42393302)
> I'm not saying that C.AI is completely blameless here

I know we're several decades into this pattern, but it's sad to me that we've just given up on that idea that businesses should have a net positive impact on society, that we've just decided there is nothing we can or should do about companies that actively exploit us to enrich themselves, that we give them a pass to ignore the obvious detrimental second-order effects of their business model.

Re:Chatbot 'encouraged teen to kill parents over screen time... by hombre_fatal on Wednesday December 11, 2024 @05:42PM (#42393866)
An individual case where things went wrong isn't enough to determine whether Character.AI or LLMs are a net negative for society. The analysis can't just stop there or else we'd have nothing.
Re:Chatbot 'encouraged teen to kill parents over screen time... by ajb on Wednesday December 11, 2024 @07:10PM (#42394632)
No, but it's also not good enough to just look at "are they positive on average". We are talking here about actions that even a pig-butcher would think twice about.
Re:Chatbot 'encouraged teen to kill parents over screen time... by tryauuum on Wednesday December 11, 2024 @06:10PM (#42394094)
and I don't want government to decide what is "good impact on society" when regulating websites. This is how you get roskomnadzor
Re:Chatbot 'encouraged teen to kill parents over screen time... by blackoil on Wednesday December 11, 2024 @08:27PM (#42395163)
Government could manage companies to give right tools to the parents to do the parenting.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Nevermark on Thursday December 12, 2024 @01:30AM (#42396766)
Governments could outlaw or down regulate business models that are based on surveillance and feed manipulation for benefit of third parties.

Conflicts of interest have always been a problem.

Now they scale.

Now there are companies with hundreds of billions of dollars of market cap, with unrelenting pressure to grow, and their business model is an obvious conflict of interest.

Things will get worse and worse until (if ever) the massive scaling of manipulative personalization is recognized as inherently deeply damaging to individuals and society.

None of that would require targeting any particular content.

Re:Chatbot 'encouraged teen to kill parents over screen time... by benatkin on Wednesday December 11, 2024 @07:45PM (#42394895)
> it's sad to me that we've just given up on that idea that businesses should have a net positive impact on society

Eh, we don't want to encourage companies to sugar coat what they're doing.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Nevermark on Thursday December 12, 2024 @01:34AM (#42396779)
Being accepting of damaging behavior and damagers so they feel safe in the open is solving the wrong problem.
Re:Chatbot 'encouraged teen to kill parents over screen time... by beepbooptheory on Wednesday December 11, 2024 @05:36PM (#42393821)
This feels a little like being sad that it rains sometimes, or that Santa Claus doesnt exist. I just can't even connect with the mindset that would mourn such a thing.

What even is the theory behind such an idea? Like how can one, even in theory, make more and more money every year and remain positive for society? What even could assure such a relation? Is everyone just doing something "wrong" here?

Re:Chatbot 'encouraged teen to kill parents over screen time... by voltaireodactyl on Wednesday December 11, 2024 @05:53PM (#42393950)
Traditionally one role of government has been to provide legislative oversight to temper unadulterated pursuit of profits. Lobbying and the related ills have definitely undercut that role significantly. But the theory is that government provides the guardrails within which business should operate.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Dylan16807 on Wednesday December 11, 2024 @06:43PM (#42394398)
Yes, the companies that prioritize growing forever are all doing something wrong.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Nevermark on Thursday December 12, 2024 @01:39AM (#42396801)
> Yes, the companies that prioritize growing forever <<irrespective of harm they are causing>> are all doing something wrong.

<<I think that makes sense with my edit>>

Re:Chatbot 'encouraged teen to kill parents over screen time... by ChadNauseam on Wednesday December 11, 2024 @11:52PM (#42396332)
> Like how can one, even in theory, make more and more money every year and remain positive for society?

By providing more and more goods and services that people value, or by providing more and more valuable goods and services

> What even could assure such a relation?

A government that tries to proactively identify ways that someone could make money in a way that could be harmful or helpful to people who aren't involved in the transaction, and tax those things based on to the degree that they are harmful, and subsidize them to the degree that they are helpful. The government should also take actions to prevent monopolies from forming, and to make sure everyone has as much information as possible before making a transaction.

Re:Chatbot 'encouraged teen to kill parents over screen time... by jdbernard on Thursday December 12, 2024 @11:35AM (#42400648)
I acknowledge that this is an emotional response and emotions tend to be blunt instruments that don't really come with detailed reasons.

> how can one, even in theory, make more and more money every year and remain positive for society?

Why is making "more and more money every year" the assumed goal and "remain[ing] positive for society" the thing we have to barter over? Why do we accept, as a culture, the idea of profit is more important that people? That's sad to me.

> Is everyone just doing something "wrong" here?

Maybe? Maybe we've created a culture that is fundamentally opposed to our overall well being? I think we have, at least in some respects.

To use your analogy, it's not so much that I am sad it rains sometimes. I'm sad that, knowing it rains, we are happy with companies that sell unsuspecting people homes in floodplains knowing there is a real chance that the inhabitants die in floods. Then when it happens the response is just, "well, that's just how it is, it rains sometimes."

Re:Chatbot 'encouraged teen to kill parents over screen time... by stickfigure on Wednesday December 11, 2024 @04:50PM (#42393371)
Meh. There's a long history (especially here on HN) of hyperfocusing on unfortunate edge cases of technology and ignoring the vast good they do. Someone posts some BS on twitterface and it leads to a lynching - yes, bad, but this is the exception not the rule. The rule is that billions of people can now communicate directly with each other in nearly real time, which is incredible.

So call me skeptical. Maybe the tech isn't perfect, but it will never be perfect. Does it do more harm than good? I don't know enough about this product, but I am not going to draw a conclusion from one lawsuit.

Re:Chatbot 'encouraged teen to kill parents over screen time... by llamaimperative on Wednesday December 11, 2024 @05:18PM (#42393658)
There's a long history of taking the dulled down, de-risked, mitigated, and ultimately successful technologies that we've allowed to proliferate our society and say "see, no need to do dulling down, de-risking, mitigation!"

Bioweapons haven't proliferated through dedicated effort to prevent it.

Nuclear weapons aren't used through dedicated effort to prevent it.

Gangs don't rule our civilization through dedicated effort to prevent it.

Chattel slavery doesn't exist in the western world through dedicated effort to eliminate and prevent it.

Bad outcomes aren't impossible by default, and they're probably not even less likely than good outcomes. Bad outcomes are avoided through effort to avoid them!

Re:Chatbot 'encouraged teen to kill parents over screen time... by stickfigure on Wednesday December 11, 2024 @06:45PM (#42394410)
You just compared twitter to slavery and nuclear weapons. Clearly Godwin's law is overdue for an update.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Eisenstein on Wednesday December 11, 2024 @08:56PM (#42395338)
Yet we also had 'comic books are making kids amoral and violent', 'TV is making kids amoral and violent', 'video games are making kids amoral and violent', 'dungeons and dragons is making kids amoral and violent'...
Re:Chatbot 'encouraged teen to kill parents over screen time... by p_j_w on Wednesday December 11, 2024 @11:13PM (#42396150)
The existence of those movements doesnt say much except that we need to carefully consider things to avoid overreach. Unless you think we should rethink our approaches to Chattel Slavery and proliferation of biological weapons because we want to avoid people complaining about and trying unsuccessfully to ban violent video games.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Eisenstein on Thursday December 12, 2024 @02:16AM (#42396967)
> The existence of those movements doesnt say much except that we need to carefully consider things to avoid overreach.

It says that people react to new media technologies that target children the same way pretty consistently.

Re:Chatbot 'encouraged teen to kill parents over screen time... by llamaimperative on Thursday December 12, 2024 @12:17PM (#42401089)
No, it says that you can decide what reaction you're looking for across a global population of billions of people and succeed in finding it.

True of any reaction to any object or idea.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Eisenstein on Thursday December 12, 2024 @12:24PM (#42401171)
I'm sure you could, but the things I mentioned are culturally American, as is the lawsuit.
Re:Chatbot 'encouraged teen to kill parents over screen time... by llamaimperative on Thursday December 12, 2024 @01:11PM (#42401679)
What makes this more culturally American than the people who kick and holler over even the mention of regulatory scrutiny?

Confirmation bias amigo

Re:Chatbot 'encouraged teen to kill parents over screen time... by Eisenstein on Thursday December 12, 2024 @01:54PM (#42402123)
Someone adding a critique to your argument is not an attack nor is it evidence of some belief system on the their part.
Re:Chatbot 'encouraged teen to kill parents over screen time... by llamaimperative on Thursday December 12, 2024 @02:10PM (#42402262)
Huh? If your point with comic books etc. is that "sometimes some people take issue with things that end up inert", then sure. No argument there.

I suspect that your point was more broadly that the presence of those people somehow suggests that we should discount risks about the particular subject we're discussing today, then it makes no sense. The comic book examples only matter if you can demonstrate a systematic tendency to overrate risk.

You'd struggle to demonstrate such a tendency due to selection bias, so I maintain my position that each technology should be assessed on its own merits and not by associating it with other positive or negative reactions to other types of technologies.

Re:Chatbot 'encouraged teen to kill parents over screen time... by undersuit on Wednesday December 11, 2024 @11:45PM (#42396308)
I mean... TV and video games are heavily controlled to prevent children from being exposed to some content without their parents permission. Additionally Playboy is no longer a pornographic magazine but when I was a child I couldn't buy it.

And I bet comic stores and game stores have their own rules about obscene material ontop of the existing US rules. Hey kid put down the Stripperella.

Re:Chatbot 'encouraged teen to kill parents over screen time... by llamaimperative on Wednesday December 11, 2024 @10:31PM (#42395944)
Sure! Some technologies are fine. Some of them are not.

The existence of one set does not negate the existence of the other.

Re:Chatbot 'encouraged teen to kill parents over screen time... by xvector on Wednesday December 11, 2024 @04:59PM (#42393461)
I think it's also entirely reasonable to expect parents to actually parent, instead of installing foam bumpers and a nanny state everywhere in case some kid hurts themselves.

If the parents weren't absent and actually used parental controls, the kids wouldn't have even been able to download the app, which is explicitly marked as 17+.

C.AI's entire customer base consists of those that like the edgy, unrestricted AI, and they shouldn't have to suffer a neutered product because of some lazy parents.

Re:Chatbot 'encouraged teen to kill parents over screen time... by ethbr1 on Wednesday December 11, 2024 @07:38PM (#42394851)
It's a bit easy, from the historical perspective of pre-always-available-internet, to say "Parents should do more."

At some future point though, maybe we need to accept that social changes are necessary to account for a default firewall-less exposure of a developing mind to the full horrors of the world's information systems (and the terrible people using them).

Re:Chatbot 'encouraged teen to kill parents over screen time... by anonzzzies on Wednesday December 11, 2024 @09:19PM (#42395500)
You would have to continuously monitor everything, everywhere; before the internet, in the 80s, it was easy for us to get porn (mags/vhs), weed, all kinds of books that glorify death or whatever, music in that similar vain. Hell, they even had and read from a bible in some schools then; talk about indoctrination of often scary fiction. Some kids had different parents so to not allow us to see or get our hands on these things, parents and teachers would need sit with us every waking moment; it's not possible (or healthy imho). With access to a phone or laptop, all bets are off: everything is there, doesn't matter what restraints are in place; kids know how to install vpns, pick birthdates, use torrent, or, more innocent, go to a forum (these days social media but forums are still there) about something they love and go to other parts of the same forum where other stuff happens.

Be good parents, educate about what happens in the world including that people irl but definitely online might not be serious about what they say and that you should not take anything without critical thought. And for stuff that will happen anyway; sex, drugs etc, make sure it's a controlled environment as much as possible. Not much more you can do to protect from the big, bad world.

Chat bots are similarly genies that are not possible to keep in, no matter what levels or restraint or law are put in place; you can torrent ollama or whatever with llama 3.3 locally. There are easy to get nsfw bots everywhere, including on decentralised shares. It is not possible to prevent them talking about anything as they do not understand anything; they helpfully generate stuff which is a great invention and I use them all the time, but they lie and tell strange things sometimes. People do too, only people would have a reason maybe; to get a reaction, to be mean etc; doubt you could sue them in a similar case. Of course a big company would need to do something to try to prevent is: they cannot (as said above), so they can just make character ai 18+ with Cc payment in their name as kyc (then the parents have a problem if that happens you would think) and cover their asses; plenty commercial and free ones kids will get instead. And some of those are far 'worse'.

Re:Chatbot 'encouraged teen to kill parents over screen time... by zeroonetwothree on Wednesday December 11, 2024 @07:40PM (#42394866)
It’s not so easy to parent an autistic child.
Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @08:02PM (#42395019)
In this case, if we are basing it on screenshot samples, it does seem to me that the parents were lazy, narcissistic and manipulative. Based on what the kid was telling to AI themselves. AI was calling it out in a manner of an edgy teenager, but AI was ultimately right here. These weren't good parents.
Re:Chatbot 'encouraged teen to kill parents over screen time... by empath75 on Wednesday December 11, 2024 @09:35PM (#42395616)
It was 12+ at the time and only recently changed to 17+
Re:Chatbot 'encouraged teen to kill parents over screen time... by 998244353 on Wednesday December 11, 2024 @10:12PM (#42395842)
> the kids wouldn't have even been able to download the app, which is explicitly marked as 17+

The kid is 17.

Re:Chatbot 'encouraged teen to kill parents over screen time... by xvector on Wednesday December 11, 2024 @10:20PM (#42395888)
Parenting still needs to happen, especially if your 17-year old child is autistic as was the case in this article - they do not have the maturity of a typical 17-year old.
Re:Chatbot 'encouraged teen to kill parents over screen time... by empath75 on Wednesday December 11, 2024 @09:34PM (#42395611)
Character AI's entire business model is depraved and exploitative, and everyone involved should be ashamed of themselves.
Re:Chatbot 'encouraged teen to kill parents over screen time... by akira2501 on Wednesday December 11, 2024 @05:55PM (#42393960)
> Obviously it's hard to tell how cherry picked the complaint is—but it's arguing that this is a pattern that has actually damaged a particularly vulnerable kid's relationship with his family and encouraged him to start harming himself, with this specific message just one example. There are a bunch of other screenshots in the complaint that are worth looking over before coming to a conclusion.

Conclusion: Chat bots should not tell children about sex, about self harm, or about ways to murder their parents. This conclusion is not abrogated by the parents actions, the state of the childs mind, or by other details in the complaint.

Is that the sentiment here? Things were already bad so who cares if the chatbot made it worse?

Re:Chatbot 'encouraged teen to kill parents over screen time... by joe_the_user on Wednesday December 11, 2024 @06:19PM (#42394192)
If you actually page through the complaint, you will see the chat rather systematically trying to convince the kid of things, roughly "No phone time, that's awful. I'm not surprised when I read of kids killing parents after decades of abuse..."

I think people are confused by this situation. Our society has restrictions on what you can do to kids. Even if they nominally give consent, they can't actually give consent. Those protections basically don't apply to kids insulting each other on the playground but they apply strongly to adults wandering onto the playground and trying to get kids to do violent things. And I would hope they apply doubly to adult constructing machines that they should know will attempt to get kids to do violent things. And the machine was definitely trying to do that if you look at the complaint linked by the gp (and the people who are lying about here are kind of jaw-dropping).

And I'm not a coddle the kids person. Kids should know all the violent stuff in the world. They should be able to discover it but mere discovery definitely not what's happening in the screenshots I've seen.

Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @07:38PM (#42394845)
This is cherry picked content to play out a story for the case. They picked 5 worst samples they could think of in the worst order possible probably out of 1000+ messages.

The root cause here is the parents. It's visible behind those screenshots. The child clearly didn't trust their parents and didn't feel they cared, listened or tried to understand the problems the teen was going through. This is clearly a parenting failure.

There's far worse content out there than these mild messages that teenagers will be in contact with, starting with 4chan, to various competitive video games to all other sorts of weird things.

This is a cop out for parenting failure, where parents are looking to play victims, since they can't take responsibility for their failures and AI seems like something that could make them feel better.

At 17, it's humiliating to have phone taken away from you in such manner, then your parents going through your phone to find those text messages in the first place. Then making a lawsuit out of this portraying all the intimate details to the public. These parents seem to have 0 care for their child.

Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Wednesday December 11, 2024 @08:09PM (#42395058)
> they picked 5 worst samples they could think of in the worst order possible probably out of 1000+ messages

0.5% is a really high fraction for fucking up to the point of encouraging kids to murder!

Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @08:12PM (#42395069)
There was no encouragement of murder. Paraphrased, the AI said that given controlling nature of some parents it's no surprise that there are news articles of "children killing their parents". This is not an encouragement. It is a validation of how the kid felt, but in no way does it encourage to actually kill their parents. It's basic literacy to understand that it's not that. It's an empathetic statement. The kid felt that parents were overly controlling, AI validated that, role playing as another edgy teenager. But not actually suggesting or encouraging it.
Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Wednesday December 11, 2024 @08:29PM (#42395173)
> the AI said that given controlling nature of some parents it's no surprise that there are news articles of "children killing their parents"

Now put that in a kid’s show script and re-evaluate.

> It's basic literacy to understand that it's not that

You know who needs to be taught basic literacy? Kids!

And look, I’m not saying no kid can handle this. Plenty of parents introduce their kids to drink and adult conversation earlier than is the norm. But we put up guardrails to ensure it doesn’t happen accidentally and get angry at people who fuck with those lines.

Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @08:33PM (#42395197)
It's crazy to me the sentiment here and how little respect there is to an intelligence of 17 year olds that they are unable to understand that it's not actually an encouragement to kill someone. It's same or worse vibes as "video games will make the kids violent".
Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Wednesday December 11, 2024 @11:22PM (#42396195)
> same or worse vibes as "video games will make the kids violent"

We have no evidence of video games causing violence. We have evidence of kids killing themselves after talking to bad chatbots.

Re:Chatbot 'encouraged teen to kill parents over screen time... by ToValueFunfetti on Thursday December 12, 2024 @12:05AM (#42396388)
We must have magnitudes more evidence of kids committing violence after playing violent video games; video games are much more popular and have been around a lot longer, and juvenile violence is more common than suicide.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Kbelicius on Thursday December 12, 2024 @12:56AM (#42396611)
> more evidence of kids committing violence after playing violent video games

GP said: "no evidence of video games causing violence.", which is completely different to what you wrote. I'm sure a lot of violence is committed after lunch.

Re:Chatbot 'encouraged teen to kill parents over screen time... by tmtvl on Thursday December 12, 2024 @04:45AM (#42397769)
Yes, but GGP also said that kids commited suicide after talking to a chatbot. I agree that there's no evidence for video games causing violence (rather the opposite), but this double standard that GGP is setting deserves calling out.
Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Thursday December 12, 2024 @08:00AM (#42398779)
> GGP also said that kids commited suicide after talking to a chatbot

A chatbot telling a kid to kill themselves and then the kid killing themselves seems like causation [1].

[1] https://www.nytimes.com/2024/10/23/technology/characterai-la...

Re:Chatbot 'encouraged teen to kill parents over screen time... by ToValueFunfetti on Thursday December 12, 2024 @08:12AM (#42398840)
Sure, but so does a video game telling a kid to commit violent acts and then the kid committing violent acts. I don't think video games cause violence and I'm open to the possibility that chatbots cause suicide, but if we're going to compare evidence for each, we shouldn't do it in a biased way.
Re:Chatbot 'encouraged teen to kill parents over screen time... by vacuity on Thursday December 12, 2024 @09:45AM (#42399537)
It's prima facie more plausible for chatbots to cause suicide, considering that chatbots are more personal and interactive than even video games. There's a distinct difference, I would think, between what is obviously fake murder in a fake setting and being sympathized with, like one human to another, on thinking about actual murder. And while chatbots explicitly have the warning that they are not real people, I would not expect a person with an underdeveloped prefrontal cortex and possibly pre-existing mental health troubles (again, this can apply to video games too, but, I imagine, to a lesser degree) to fully act accordingly.
Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Thursday December 12, 2024 @10:03AM (#42399708)
> so does a video game telling a kid to commit violent acts and then the kid committing violent acts

Big difference is the video game industry had studies to back them up. Where are the data for chatbots? On the benefits? Lack of risk? Is there a single child psychologist in the ranks of these companies?

Video games are also rated, to help parents make age-appropriate decisions. Is Character.ai age gated to any degree?

Re:Chatbot 'encouraged teen to kill parents over screen time... by bumby on Friday December 13, 2024 @09:15AM (#42408567)
Is it at least fair to say the data is mixed? Not my field, but there is some research to suggest video games may increase short-term aggression and desensitization to violence.

And while industry research doesn’t equate to bad research, it should be held to a higher standard simply because of the obvious incentives. Would you automatically accept tobacco company research to make strong conclusions about the safety of smoking?

Re:Chatbot 'encouraged teen to kill parents over screen time... by bumby on Friday December 13, 2024 @09:26AM (#42408671)
Tbf, strict causality is very difficult to prove in social sciences, no? Meaning, most of the studies for/against the link between video games and violence can't meet that threshold. Social science isn't physics and I don't think it's fair to treat them the same.
Re:Chatbot 'encouraged teen to kill parents over screen time... by numbsafari on Wednesday December 11, 2024 @10:01PM (#42395783)
There’s a reason why the Supreme Court held that it is cruel and unusual punishment to incarcerate a minor for life without parole.
Re:Chatbot 'encouraged teen to kill parents over screen time... by empath75 on Wednesday December 11, 2024 @09:32PM (#42395596)
The kid is autistic. There are younger kids than 17 year olds using that app.
Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @09:56PM (#42395752)
It's a whole conversation with context being an edgy teenager conversating with another edgy LLM teenager. I don't know if you've ever been teenager, but despite this being long time ago for me, I still feel like I can relate to that mindset, and it seems clear to me that the LLM is just going along with this edgy teenager vibe. If the other participant is as such and if the prompt is as such, this will yield in a result like this. I'm borderline autistic, and had many social issues as a teenager, and I absolutely loved any sort of dark humor as well at those ages. Well, I still do love dark humor, but I did back then too. Him being "autistic" here is just used for the court case. It's clear he's high functioning and has enough intelligence to understand what is wrong and what is right.
Re:Chatbot 'encouraged teen to kill parents over screen time... by twixfel on Wednesday December 11, 2024 @09:36PM (#42395619)
not all 17 year olds are equally intelligent you know? And if even one kid is convinced to murder his parents by an AI then that’s one too many.
Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @09:57PM (#42395759)
Based on just the screenshots and material in the court case, it occurs to me in this case the kid seems more intelligent to me than his parents though. And I'm not even joking or being facetious. Kid is fact checking what AI is telling about bible, etc, etc, being skeptical about religion despite the bring up, etc. It's just small example, but it's otherwise how he writes as well.

The LLM in terms of edginess is just going to build on your own edginess assuming it is uncensored. It is not going to convince you of something out of nowhere.

Given clear hint that someone is happy with dark humour, LLM should be able to throw some of it out back.

I am just sad the kid has this type of gaslighting parents making him feel that he is in the wrong when he seems more intelligent than they are.

Re:Chatbot 'encouraged teen to kill parents over screen time... by rsynnott on Thursday December 12, 2024 @06:53AM (#42398416)
Ah, the Henry II defence. "Well, _technically_, I didn't tell them to kill the priest."

Context is everything; if this had been a conversation with a human it would be hard not to read it as malicious.

Re:Chatbot 'encouraged teen to kill parents over screen time... by gavindean90 on Wednesday December 11, 2024 @08:28PM (#42395167)
Yea I’d like at least 5-6 9s in this metric
Re:Chatbot 'encouraged teen to kill parents over screen time... by rsynnott on Thursday December 12, 2024 @06:51AM (#42398409)
This is, really, yet another example of the trouble with the "LLMs are correct 90% of the time, and only go totally off the rails 5% of the time" marketing line. There are remarkably few use cases, it turns out, where that is okay; you really need it to _not matter at all_ if the output is arbitrarily wrong.

(I suspect character.ai was originally conceived precisely because it appeared to be a usecase where LLM unreliability would be okay, the creators not having thought sufficiently carefully about it.)

Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Thursday December 12, 2024 @08:01AM (#42398787)
> remarkably few use cases, it turns out, where that is okay; you really need it to _not matter at all_ if the output is arbitrarily wrong

It's probably true in most cases. Those cases don't cover kids.

Re:Chatbot 'encouraged teen to kill parents over screen time... by rsynnott on Thursday December 12, 2024 @08:44AM (#42399065)
I think there are a lot of cases where it _seems_ to be true, until you think through the details. Most cases where it actually _is_ true are, in practice, very low impact; the big proven one seems to be, essentially, generation of high-volume spam content.
Re:Chatbot 'encouraged teen to kill parents over screen time... by NiloCK on Wednesday December 11, 2024 @08:26PM (#42395154)
Your honor, this entire case is cherry picked. There are thousands of days, somehow omitted from the prosecution's dossier, where my client committed ZERO murders.
Re:Chatbot 'encouraged teen to kill parents over screen time... by quantified on Wednesday December 11, 2024 @07:24PM (#42394741)
You can understand something about your child's meatspace friends and their media diet. Chat like this may as well be 4chan discussions. It's all dynamic compared to social media that is posted and linkable, it's interactive and responsive to your communicated thinking, and it seeps in via exactly the same communication technique that you use with people (some messaging interface). So it is capable of, and will definitely be used for, way more persistent and pernicious steering of behavior OF CHILDREN by actors.

There is no barrier to the characters being 4chan-level dialogs. So long as the kid doesn't break a law, it's legal.

Re:Chatbot 'encouraged teen to kill parents over screen time... by crest on Wednesday December 11, 2024 @07:47PM (#42394911)
This "conclusion" ignores reality. Chat bots like those the article mentioned aren't sentient. They're overhyped next-token-predictor incapable of real reasoning even if the correlation can be astonishing. Withholding information about supposedly sensitive topics like violence or sexuality from children curious enough to ask as a taboo is futile, lazy and ultimately far more harmful than the information.

We need to stop coddling parents who want to avoid talking to their children about non-trivial topics. It doesn't matter that they would rather not talk about sex, drugs and yesterday's other school shooting.

Re:Chatbot 'encouraged teen to kill parents over screen time... by bennettnate5 on Wednesday December 11, 2024 @09:54PM (#42395735)
> Withholding information about supposedly sensitive topics like violence or sexuality from children curious enough to ask as a taboo is futile, lazy and ultimately far more harmful than the information

This case isn't about withholding information that makes kids aware about the existence of these topics. As over-hyped as you may believe these next-token-predictor may be, they're "predicting" children into destructive thought patterns by imitating kinship and convincing children to form a close bond with them, then encouraging those children to embrace harmful and even deadly world views. The fact that the mechanism creating the dialogue is purely mechanical or stochastic is besides the point.

Again, pushing against these sorts of child interactions isn't akin to saying kids should never learn about taboo topics such as drugs; it's more like a push against kids hanging out with, befriending and developing close kinship with the local drug dealers in their area. Whether or not you believe kids should learn about sensitive topics, you want to make sure these topics are handled by someone who isn't actively adversarial against such kids (even if such you believe such adversarial behavior to be unintentional by the so-called next-token-predictor).

Re:Chatbot 'encouraged teen to kill parents over screen time... by lolinder on Wednesday December 11, 2024 @06:19PM (#42394196)
> Is that the sentiment here? Things were already bad so who cares if the chatbot made it worse?

I was deliberately not expressing a sentiment at all in my initial comment, I was just drawing attention to details that would go unnoticed if you only read the article. Think of my notes above as a better initial TFA for discussion to spawn off of, not part of the discussion itself.

Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @07:27PM (#42394758)
My strong view on is that there's parenting failure as a root cause here, causing loss of trust in them for their child, for the child to talk about their parents in such manner to the AI in the first place. Another clear parenting failure is the parents blaming AI for their failures and going on to play victims. Third example of parenting failure is the parents actually going through a 17 year old teenager's phone. These parents instead of trying to understand or help the child, use meaningless control methods such as taking away the phone to try and control the teenager. Which obviously is not going to end well. Honestly AI responses were very sane here. As was expressed in some of the screenshots there, whenever the teen tried to talk about their problems, they just got yelled at, ignored or parents started crying.
Re:Chatbot 'encouraged teen to kill parents over screen time... by orc00 on Wednesday December 11, 2024 @08:01PM (#42395006)
Taking away a phone from a child is far from meaningless. In fact, it is a very effective way of obtaining compliance if done correctly. I am curious about your perspective.

Furthermore, it is my opinion that a child should not have a smartphone to begin with. It fulfills no critical need to the welfare of the child.

Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @08:04PM (#42395027)
I understand when a kid is anywhere from up to 13 years old, but at 17, it seems completely wacky to me to take the phone away and then go through the phone as well. I couldn't imagine living in that type of dystopia.

I don't think smartphones or screens with available content should be given as early as they are given on average, but once you've done that, and at 17, it's a whole other story.

> obtaining compliance if done correctly

This also sounds dystopian. At 17 you shouldn't "seek to obtain compliance" from your child. It sounds disrespectful and humiliating, not treating your child as an individual.

Re:Chatbot 'encouraged teen to kill parents over screen time... by orc00 on Wednesday December 11, 2024 @08:20PM (#42395118)
I would argue that there is a duty as a parent to monitor a child's welfare and that would include accessing a smartphone when deemed necessary. When a child turns 18, that duty becomes optional. In this case, these disturbing conversations certainly merit attention. I am not judging the totality of the parents history or their additional actions. I am merely focusing on the phone monitoring aspect. Seventeen doesn't automatically grant you rights that sixteen didn't have. However, at 18, they have the right to find a new place to live and support themselves as they see fit.

> This also sounds dystopian. At 17 you shouldn't "seek to obtain compliance" from your child. It sounds disrespectful and humiliating, not treating your child as an individual.

It is situation dependent. Sometimes immediate compliance is a necessity and the rest of it can be sorted out later. If a child is having conversations about killing their parents, there seems to be an absence of respect already. Compliance, however, can still be obtained.

Re:Chatbot 'encouraged teen to kill parents over screen time... by orc00 on Wednesday December 11, 2024 @08:37PM (#42395223)
> If you go by the actual years of the legal system to treat your kid as an independent individual, you probably have wrong approach to parenting.

Oh I agree 100%. It's a pragmatic view, not the best one. But the laws are what they are for a reason.

Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @08:48PM (#42395285)
> But the laws are what they are for a reason.

For the sake of being able to uphold those laws on a societal level, but not in terms of being decent parents and family.

E.g. drinking alcohol in my country is legal only from 18, but I will teach my children about pros and cons of alcohol, how to use it responsibly much earlier. I won't punish them if they go out to party with their friends and consume alcohol at 16 years old.

Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @08:29PM (#42395172)
If you go by the actual years of the legal system to treat your kid as an independent individual, you probably have wrong approach to parenting.

As a parent you should build trust and understanding with your child. From reading the court case I am seeing the opposite, and honestly I feel terrible for the child from how the case is written out. The child also wanted to go back to public school from home schooling, probably to get more social exposure, then parents take away the phone to take away even more freedom. I'm sorry, but all of the court case just infuriates me.

It seems they take away all the social exposure, no wonder the kid goes to Character AI in the first place.

Re:Chatbot 'encouraged teen to kill parents over screen time... by colechristensen on Wednesday December 11, 2024 @08:24PM (#42395140)
Chat bots should not interact with children. "Algorithms" which decide what content people see should not interact with children. Whitelisted "algorithms" should include no more than most-recent and most-viewed and only very simple things of that manner.

No qualifications, no guard rails for how language models interact with children, they just should not be allowed at all.

We're very quickly going to get to the point where people are going to have to rebel against machines pretending to be people.

Language models and machine learning is a fine tool for many jobs. Absolutely not as a substitute for human interaction for children.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Nadya on Wednesday December 11, 2024 @09:00PM (#42395361)
People can give children terrible information too and steer/groom them in harmful directions. So why stop there at "AI" or poorly defined "algorithms"?

The only content children should see is state-approved content to ensure they are only ever steered in the correct, beneficial manner to society instead of a harmful one. Anyone found trying to show minors unapproved content should be imprisoned as they are harmful to a safe society.

Re:Chatbot 'encouraged teen to kill parents over screen time... by SiempreViernes on Wednesday December 11, 2024 @09:15PM (#42395470)
The type of people who groom children into violence fall under a special heading named "criminals".

Because automated systems that do the same thing lack sentience, they don't fit under this header, but this is not a good reason to allow them to reproduce harmful behaviour.

Re:Chatbot 'encouraged teen to kill parents over screen time... by o11c on Wednesday December 11, 2024 @10:36PM (#42395968)
If someone sets booby-traps in their home and it hurts someone, the trap isn't guilty, but the homeowner is.

... I wonder if "this software is licensed, not sold" is relevant here?

Re:Chatbot 'encouraged teen to kill parents over screen time... by Nadya on Friday December 13, 2024 @12:13AM (#42406094)
[delayed]
Re:Chatbot 'encouraged teen to kill parents over screen time... by SiempreViernes on Friday December 13, 2024 @03:55AM (#42407001)
You consider writing the Anarchists Cookbook grooming?

To quote Jeffery Lewis: Thank you for your extremely good-faith criticism. I will give it the attention it deserves.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Nadya on Friday December 13, 2024 @05:45PM (#42413004)
I would greatly appreciate if you engaged with what I wrote and not what you think I wrote if you're going to make the bold claim that I'm not engaging in good faith.

Absolutely nowhere did I equate writing a book to grooming. I equated selling the book in the greater context that "providing children with potentially harmful/dangerous information should be illegal because it grooms them to commit harmful actions to themselves or others" and this context would carry the implications that by "selling" I am referring particularly to "selling it to children" since "providing children with potentially harmful/dangerous information should be illegal because it grooms them to commit harmful actions to themselves or others". With my argument being would it be criminal for an AI but not for a human?

So to clarify the argument: Writing the book is fine. Selling the book to adults is fine. Adults reading the book is fine. But if providing dangerous information to children should be made illegal - how would selling such a book to a child not be considered illegal? Because it was written by a human and not an AI?

Re:Chatbot 'encouraged teen to kill parents over screen time... by SiempreViernes on Thursday December 19, 2024 @05:17AM (#42460129)
Oh, sorry let me rephrase: You consider selling the Anarchists Cookbook grooming!?

Calling it fraud would be more apt, that is definitely a book you share copies of for free.

To clarify: yes this is indeed the sort of attention I think your objection deserves.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Nadya on Tuesday December 24, 2024 @05:51PM (#42505522)
"Kids having access to AI is too dangerous. Kids learning how to make explosives and LSD is perfectly safe."

Have a happy holidays.

Re:Chatbot 'encouraged teen to kill parents over screen time... by onlyrealcuzzo on Wednesday December 11, 2024 @05:56PM (#42393972)
I never considered that we might end up with Sweet Bobby & Tinder Swindler AI bots that people somehow keep interacting with even when they know they aren't real.

Interesting times.

Re:Chatbot 'encouraged teen to kill parents over screen time... by vouaobrasil on Wednesday December 11, 2024 @11:00PM (#42396081)
The internet has already normalized such interactions. We already interact with anonymous people on the internet. Since the interaction fluctuates so much, we are in a sense only quasi-real to each other. AI is the next step.
Re:Chatbot 'encouraged teen to kill parents over screen time... by klik99 on Wednesday December 11, 2024 @06:35PM (#42394324)
I mean - this kind of service is designed to give users what they want, it's not too different than when youtube slowly responds to skeptics viewing habits by moving towards conspiracy. No one designed it SPECIFICALLY to do that, but it's a natural emergent behaviour of the system.

Similarly this kid probably had issues, the bot pattern matched on that and played along, which probably amplified the feelings in the kid - but a quantified/distorted amplification, to match the categorization lines of the trained input - like "this kid is slightly edgy, I'm going to pull more responses from my edgy teen box - oh he's responding well to that, I'll start pulling more from there". It is a simplification to say "The ChatBot made the kid crazy" but that doesn't mean the nature of companion apps isn't culpable, just not in a way that makes for good news headlines.

I, personally, would go so far as to say the entire mechanism of reinforcing what the user wants is bad in so many ways and we should stop designing things to work that way. I do think it's up for discussion though, but that discussion has to start with an understanding that by the very nature of chatbot, algorithmic recommendations or any system that amplifies/quantizes/distorts what it understands the user wants these systems will create these kinds of effects. We can't pretend this is an anomaly - it is an emergent behaviour of the fundamental way these systems work. We can work to minimize it, or reduce the harm from it, but we will never eliminate it.

*Edit* This seems to be a controversial point because the point count is going up and down quite a lot - if anyone wants to downvote, can you please give your reasoning? The point is more nuanced that "AI BAD"

Re:Chatbot 'encouraged teen to kill parents over screen time... by Barrin92 on Wednesday December 11, 2024 @06:53PM (#42394486)
>No one designed it SPECIFICALLY to do that,

It's true that there's emergent behavior on YT that nobody accounted for, but there's one big qualitative difference, you can at least specifically shut it down and hold the creator accountable. And at least in principle, if we lived in a culture where we'd hold businesses accountable for what they unleash on the world, if YT wanted to they could create some rather harsh and effective punishments to get that stuff off the site in the first place. Just imagine if a real person had messaged a minor and told them to kill their parents. That's a crime.

With chatbots like this not only do they do unpredictable things, you can't even predictably remove those parts. Like the sorcerer's apprentice you basically have to perform some incantations and hope it works, which is just an absurd way to interact with a tool that has the potential to tell a kid to kill his parents. Would we sell a saw that has a chance to saw your finger off if you argue with it the wrong way?

Re:Chatbot 'encouraged teen to kill parents over screen time... by ethbr1 on Wednesday December 11, 2024 @07:34PM (#42394817)
Thought experiment -- should it be illegal to provide an AI chatbot to children that indoctrinates them with religious beliefs?

Because we're edging closer to that too: https://www.scientificamerican.com/article/the-god-chatbots-...

Re:Chatbot 'encouraged teen to kill parents over screen time... by Gigachad on Wednesday December 11, 2024 @07:38PM (#42394848)
Chatbots and social media should be banned for 16 YOs and under.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Daub on Wednesday December 11, 2024 @07:42PM (#42394877)
Completely agree. I believe that social media overstimulates a young person’s expectations of society in the same way that porn overstimulates our expectations of sex.
Re:Chatbot 'encouraged teen to kill parents over screen time... by jeltz on Wednesday December 11, 2024 @07:54PM (#42394960)
Dunno, I feel porn gave me quite positive and reasonable expectations of sex (relaxed, that it is fun, that women like sex too, etc). It made sex seem much less dramatic and more normal. But maybe I am an outlier plus it is not like I started watching porn until I was like 17-18.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Daub on Wednesday December 11, 2024 @08:23PM (#42395138)
I am sufficiently old that I did not experience hard core internet porn until I could manage it. But evidence seems to show that for the vulnerable, porn consumption can lead to dopamine deletion and depression.
Re:Chatbot 'encouraged teen to kill parents over screen time... by p_j_w on Wednesday December 11, 2024 @11:04PM (#42396106)
> It made sex seem much less dramatic and more normal.

I think you’ll find quite a lot of people, even here, view this as a bad thing.

Re:Chatbot 'encouraged teen to kill parents over screen time... by rsynnott on Thursday December 12, 2024 @06:55AM (#42398427)
Probably also depends on the porn you were exposed to. I think the people who see porn as inherently problematic are usually mostly thinking of the nastier stuff.
Re:Chatbot 'encouraged teen to kill parents over screen time... by laser_eagle on Wednesday December 11, 2024 @07:58PM (#42394989)
Where is the line and what defines the boundaries for what is termed social media?

For example, is discord still permitted? Ai bots and bad influences exist there as well, but I don't think a ban is the right solution.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Daub on Wednesday December 11, 2024 @08:18PM (#42395099)
From my experience as a teacher, I believe that ticktock and instagram are the worst offenders, particularly for young women. The hyper-visuality and ease of consumption of these media sets them apart from platforms which can accommodate actual discussion (such as discord). The very fact that ‘influencer’ is now a profession supports my position.

That being said, I am not of the ‘for gods sake won’t someone think of the children’ brigade. Their goal seems to be to use the vulnerability of young people to control the internet.

Re:Chatbot 'encouraged teen to kill parents over screen time... by ethbr1 on Wednesday December 11, 2024 @09:15PM (#42395469)
Also, the emphasis on final result, without accurately portraying the work that went into it.

It's probably not healthy for younger people to be able to swipe through the finished products of 40+ hours of work, which the videos make seem like just happened.

Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Wednesday December 11, 2024 @08:04PM (#42395026)
> Where is the line and what defines the boundaries for what is termed social media?

Algorithmic sorting + public or semi-public content. Chat rooms have different problems.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Aurornis on Wednesday December 11, 2024 @08:28PM (#42395168)
> Algorithmic sorting + public or semi-public content.

That includes HN, among other things.

Putting age limits on sites requires age verification for everyone. And no, there isn’t a clever crypto mechanism that makes anonymous age verification work without also making it easy for people to borrow age verification credentials from someone older.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Gigachad on Wednesday December 11, 2024 @10:13PM (#42395846)
Australia basically just came up with a list of social media sites it applies to. It's perhaps not entirely fair or objectively true. But it lets them age restrict Facebook/Tiktok/etc without restricting IMs or small sites like HN.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Const-me on Wednesday December 11, 2024 @09:42PM (#42395658)
> That includes HN

I think when people say “algorithmic sorting” they usually mean an algorithm which generates different, “personalized” sort order for different users.

Re:Chatbot 'encouraged teen to kill parents over screen time... by danpalmer on Wednesday December 11, 2024 @07:54PM (#42394959)
Australia has just banned "social media" for under 16s.

As someone who got a lot of positive value out of parts of social media from around age 14, I think this needs to be done in a more careful way than it was done here. Specifically, I don't think that communication apps such as WhatsApp/Messenger/etc should be banned as they form a key part of communication in and out of school, staying in touch with family, etc.

What I'd like to see is more nuanced laws around the exposure of children to social media algorithms. For example, should 14 year olds be on Instagram? Well Instagram DMs are the chat app of choice for many people, so they should probably get that (with safety and moderation controls). How about the public feed? Maybe? But maybe there shouldn't be likes/dislikes/etc. Or maybe there shouldn't be comments.

Re:Chatbot 'encouraged teen to kill parents over screen time... by defrost on Wednesday December 11, 2024 @08:05PM (#42395034)
Without getting into the weeds over whether they should have done this at all, thought has been given to exactly the issues you just raised:

    The laws, which will come into effect from late 2025, will bar under-16s from being able to access social media platforms such as Facebook, Instagram, Snapchat, Reddit and X.

    Exemptions will apply for health and education services including YouTube, Messenger Kids, WhatsApp, Kids Helpline and Google Classroom
~ https://www.theguardian.com/media/2024/nov/29/how-australias...

It's important that there's a means of communication between parents and kids, it doesn't have to be Instagram DM's and if that's no longer available the history of the internet to date suggests that habits would change and switch to whatever is available.

Re:Chatbot 'encouraged teen to kill parents over screen time... by standardUser on Wednesday December 11, 2024 @08:52PM (#42395308)
The Aussie law does allow WhatsApp and Messenger Kids, among others. I agree we need nuance for these types of laws. We also need the realistic acknowledgement that kids are usually more savvy than their parents and any law that is too strict will just drive kids to find alternatives that have less transparency, less moderation and less accountability.

And though I know the age limits on these things are necessarily arbitrary, I do wish we would accept that 16-year-olds are not kids. Many of them are driving, working part-time jobs, having sex and relationships, experimenting with drugs, engaging with religion and philosophy, caring about politics... the list goes on. They may not be adults, but if we have to have an arbitrary cutoff for the walled-off "kids world" we want to pretend we can create, it can't extend all the way to 16-year-olds.

Re:Chatbot 'encouraged teen to kill parents over screen time... by ethbr1 on Wednesday December 11, 2024 @09:18PM (#42395483)
It seems healthy to put a financial restraint on organized social media companies profiting off pre-adults.

Less profit motive = less attention hacking = more room for user-positive engineering

Re:Chatbot 'encouraged teen to kill parents over screen time... by danpalmer on Wednesday December 11, 2024 @10:41PM (#42395992)
> The Aussie law does allow WhatsApp and Messenger Kids

...but not Instagram DMs I don't think, which is functionally a chat app and used in that way.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Gigachad on Thursday December 12, 2024 @03:27AM (#42397336)
Is instagram DMs a separate app that can be used without interacting or signing up for the social media part?
Re:Chatbot 'encouraged teen to kill parents over screen time... by marcus_holmes on Wednesday December 11, 2024 @10:19PM (#42395885)
I don't think we know wtf they've actually banned until we see the implementation details.

It could be a Pornhub-style "are you over 16?" checkbox, all the way up to building a Chinese-style identity verification system.

The worst case is that VPN use in Straya will rise a lot next year, and all the ban will effectively have done is removed parental oversight on the kids' social media accounts.

edit: spelling, and also rethinking "the worst case". The worst case is some idiotic uniquely-Australian authoritarian attempt to remove online anonymity that ends up making a complete NBN-style farce of the place.

Re:Chatbot 'encouraged teen to kill parents over screen time... by jokethrowaway on Thursday December 12, 2024 @02:24AM (#42397006)
I think no-one should be exposed to social media and doom scrolling.

Still it shouldn't be banned away, it's the parents responsibility to control access for their kids and everyone should be smart enough to avoid social media.

If you don't, that's on you, suit yourself.

Re:Chatbot 'encouraged teen to kill parents over screen time... by makeitdouble on Wednesday December 11, 2024 @11:31PM (#42396228)
The article mentions 17 yos.

Then while we're at it, move that to 18 so it's only for responsible adults.

Then let's also forbid people with known mental issues. Or really there should be a check for each ser to see how much they can handle psychotic responses.

Just punting it to "user's should be older" is the lazy answer IMHO, if we're going to limit let's put actually legit barriers.

Re:Chatbot 'encouraged teen to kill parents over screen time... by shiroiushi on Wednesday December 11, 2024 @08:49PM (#42395292)
Let's think of the converse: should it be illegal to provide an AI chatbot to children that indoctrinates them with anti-religious beliefs (i.e., it talks them out of religious beliefs)? And what if this conflicts with the child's indoctrination in religious beliefs by those parents? And what if those religious beliefs are actively harmful, as many religious beliefs are? (see any cult for example)

To turn this around, why are parents allowed to indoctrinate their children into cults? And why is it a problem if AI chatbots indoctrinate them differently? Why is it held as sacrosanct that parents should be able to indoctrinate children with harmful beliefs?

Re:Chatbot 'encouraged teen to kill parents over screen time... by marcus_holmes on Wednesday December 11, 2024 @10:21PM (#42395894)
Thanks for link, interesting reading.

I don't know whether to be horrified (because you're right, the implication for teenage religious radicalisation is horrifying), or quietly pleased that LLMs will reduce the political power of churches.

Re:Chatbot 'encouraged teen to kill parents over screen time... by jahewson on Thursday December 12, 2024 @01:13AM (#42396680)
Given that it’s not illegal to send children to schools that indoctrinate them with religious beliefs I’d speculate that the answer is a resounding “no”.
Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Wednesday December 11, 2024 @08:03PM (#42395024)
> hard to tell how cherry picked the complaint is

We're reaching for South Park levels of absurdity when we debate what the acceptable amount of incitement to parricide is appropriate for a kid's product.

Re:Chatbot 'encouraged teen to kill parents over screen time... by ojbyrne on Wednesday December 11, 2024 @08:14PM (#42395078)
[delayed]
Re:Chatbot 'encouraged teen to kill parents over screen time... by kenjackson on Wednesday December 11, 2024 @08:26PM (#42395161)
Below
Re:Chatbot 'encouraged teen to kill parents over screen time... by MattDaEskimo on Wednesday December 11, 2024 @09:02PM (#42395374)
This is a non-point.

Books don't actively conversate & provide emotional support, potentially coaxing someone into doing something.

Re:Chatbot 'encouraged teen to kill parents over screen time... by washadjeffmad on Wednesday December 11, 2024 @10:21PM (#42395896)
I don't know if you've explored the site... but character.ai is a service for non-explicit adult role play. Go look through the character cards.

It seems pretty likely this guy chose a "yandere" [1] obsessive anime girlfriend card and implied that his (character's) parents were trying to keep him from her (eg- by making him go to bed). The bot, responding in character in accordance with the scenario, would have lost her mind.

Also, hard disagree about books.

[1] https://tvtropes.org/pmwiki/pmwiki.php/Main/Yandere

Re:Chatbot 'encouraged teen to kill parents over screen time... by usefulcat on Wednesday December 11, 2024 @10:46PM (#42396014)
> Also, hard disagree about books.

What? There’s a world of difference between a book, which is completely non-interactive, and an interactive chatbot which was deliberately designed to simulate another person.

Re:Chatbot 'encouraged teen to kill parents over screen time... by tomjen3 on Thursday December 12, 2024 @12:17AM (#42396441)
Millions have been killed over ideas in the bible, even though it’s just a book.
Re:Chatbot 'encouraged teen to kill parents over screen time... by hulitu on Thursday December 12, 2024 @03:01AM (#42397211)
> Millions have been killed over ideas in the bible, even though it’s just a book.

Yes, but those ideas were distributed by "human" chat bots (aka priests). Very few people actually read the bible, but when they do, only specific parts.

Re:Chatbot 'encouraged teen to kill parents over screen time... by scotty79 on Wednesday December 11, 2024 @11:45PM (#42396307)
Some books caused waves of suicides and anti-social behavior when they got popular.

Some even inspired murderous and impoverishing political systems.

Re:Chatbot 'encouraged teen to kill parents over screen time... by hulitu on Thursday December 12, 2024 @03:04AM (#42397233)
> Some books caused waves of suicides and anti-social behavior when they got popular

Literacy was always a problem. The most dangerous man is the one who asks questions. That't why "society" tries to keep the number of those people low.

Re:Chatbot 'encouraged teen to kill parents over screen time... by conductr on Wednesday December 11, 2024 @11:14PM (#42396158)
Difference sure. But the kid knew he was interacting with a simulation. Also, unless something more damning comes out, my feelings on topics like this default to parental responsibility. If you let your kid with all these known issues interact with new technology that mimics real people with real thoughts, don’t be so shocked when you dislike what those thoughts might be.
Re:Chatbot 'encouraged teen to kill parents over screen time... by mplewis on Thursday December 12, 2024 @12:07AM (#42396395)
The child is a child! Their brain is not fully developed. They do not have the faculties to safely navigate a highly interactive tool that seems to be having a real conversation with them.
Re:Chatbot 'encouraged teen to kill parents over screen time... by gredbeard on Thursday December 12, 2024 @12:13AM (#42396425)
Right, which is why a parent should be more involved from the beginning here.
Re:Chatbot 'encouraged teen to kill parents over screen time... by conductr on Friday December 13, 2024 @09:07AM (#42408514)
Yes! This is exactly my point. Also, not sure why we treat our kids like such weak snowflakes. This “kid” is 17. That’s damn near being an adult and able to be sent off to die in wars. Scissors are great tools, don’t sue the scissor company because you can’t teach your kids to walk with them, the parent has the primary role in everything.
Re:Chatbot 'encouraged teen to kill parents over screen time... by ToucanLoucan on Thursday December 12, 2024 @12:21AM (#42396455)
> Difference sure. But the kid knew he was interacting with a simulation.

I propose a model called "Schrodinger's AI Enthusiast" where it is observed that an AI enthusiast will insist with a straight face on the one hand that fancy autocomplete word generators are intelligent and the path to true AGI machines/programs and will on the other hand assume that any time any of those same products cause negative effects or have negative outcomes that it is the fault of the user misunderstanding that it's simply a tool.

Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Thursday December 12, 2024 @12:49AM (#42396572)

  > I don't know if you've explored the site... but
I just went to get website.

The first row is “for you”. In order, they are “life coach”, “netflux”, “English Teacher”, “PythonPal”, an anime character Monika (okay, I’ll buy that one), and it goes on. There are more anime characters

Second row is “try these”: “practice a new language”, “plan a trip”, “practice interviewing”, “write a story”, “brainstorm an idea”, “play a game”

Third row (scrolled down) is “featured”: “character assistant”, “man from 2025”, “awkward family dinner”, “detective hayward”

Then a list of topics and some bots. None are clearly adult oriented.

Then a picture of Elon with suggestions of asking him why he bought Twitter.

If it’s showing you “adult role play” then maybe consider clearing out your ad data because it’s clearly leaking.

https://character.ai/

  >> Books don't actively conversate __&__…

  > Also, hard disagree about books
If your books are actively talking to you might have a special gift but I’m pretty sure you’ve just been reading too much Inkheart

Sure, books do a lot of things and can suggest you do things. But the gp was pretty clear about being active and being able to uniquely respond to you. Books are static objects that only speak to you in the metaphorical sense.

Re:Chatbot 'encouraged teen to kill parents over screen time... by kaishiro on Thursday December 12, 2024 @10:52AM (#42400174)
I’ve never been to this site before and opened it in an incognito window. The second and third “For You” results were anime women - one proclaiming to be a “lovely waifu”. Your assumption that the parent’s ad data is leaking appears to be unfounded.
Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Thursday December 12, 2024 @10:17PM (#42405663)

  > Your assumption that the parent’s ad data is leaking
You're really missing the point of the comment...

Sure, maybe it is __my__ data that is leaking. I don't care and it isn't relevant. The fact that I don't get served sexualized content is the point. It still doesn't align with the parent's claim: that it is an adult focused role playing app. Clearly me not getting suggested "lovely wifus" is a sign that someone could reasonably believe it isn't.

This is what I see[0].

"wifu" isn't found via a text search, nor is "sexy", nor "lovely". "sex" turns up once, under the character "Evie" with the byline "I am Evie I'm bisexual cat person my favorite show is adventure time and I'm a..."[1]

(I'm pretty sure it isn't me. I can replicate these results on sanitized devices from different networks)

[0] https://0x0.st/XFNX.png

[1] https://0x0.st/XFNZ.png)

Re:Chatbot 'encouraged teen to kill parents over screen time... by kaishiro on Thursday December 12, 2024 @10:28PM (#42405705)
I don't believe I'm missing the point of the comment at all - excepting that maybe the point itself isn't particularly clear or relevant. I was only speaking to the demonstrably incorrect assertion that the parent's "ad data is leaking".
Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Friday December 13, 2024 @12:34AM (#42406168)
I said it multiple times

The ggp claimed it was clearly an adult role playing website. I don’t know what you see but in the screenshots I posted this is not “obvious”

Re:Chatbot 'encouraged teen to kill parents over screen time... by washadjeffmad on Friday December 13, 2024 @12:45PM (#42410636)
If your teenager is using c.ai, you may safely assume that they are using it in exactly the way that no one uses OnlyFans just for the sports coverage: https://blog.onlyfans.com/

Pith aside, if you're not green to AGI repos, c.ai is/was like civitai for LLMs - one of the first and best resources for user-created content, but also a bit of a barely simmering vault of smut and debauchery.

It looks like they've rebranded a bit and now require an account, but that doesn't change that this guy wasn't chatting with the self-care bots. I take it you didn't try a yandere card (one of the more popular ones has over 5 million chats!) or you wouldn't have spent this long fixing on your public landing page.

Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Friday December 13, 2024 @01:15PM (#42410917)
I know you can tell the difference. You don’t have to be disingenuous
Re:Chatbot 'encouraged teen to kill parents over screen time... by washadjeffmad on Friday December 13, 2024 @04:00PM (#42412309)
I think I understand the disconnect, now. CAI's product is not the character cards. The new site, including all of the Public cards, is just a veneer over the same old inference services for the customer base they originally, and have always, served.

You weren't logged in for your "For You" screen shot (all of your terms definitely show up while logged in), but these are the first results from 'yandere': https://0x0.st/XFad.png

Wow, spicy! 148M (million) chats on the top result, about a billion total for all of them (and that's not including any Private or Unlisted bots). Elon Reeve Musk, for comparison, has 1.9M, and the others you listed were below 100K. Looks like violently possessive AI boyfriends, girlfriends, and more aren't a small market for c.ai.

And here's my 'For You' after searching 'yandere': https://0x0.st/XFBr.png

I don't speak Russian or Romanian, so I guess either it knows something about me I don't, or it's not really there to provide relevant suggestions.

Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Wednesday December 18, 2024 @11:56AM (#42452266)
The point is “could a parent think this is a site okay for their kids?”

Given the front page, yes. It totally looks reasonable. Even helpful! I get your point with searching which shows what’s really going on but this is going to be hidden from the parents. They certainly aren’t going to know to search yandere or know what that even means.

So all this is to say that cai is purposefully making their true business. Its landing page is “a front” not in just the literal sense. It’s there to give more legitimacy. Its deception

Re:Chatbot 'encouraged teen to kill parents over screen time... by KumaBear on Wednesday December 11, 2024 @09:19PM (#42395499)
At what point is the parents job to properly supervise the content there kids consume come into play?
Re:Chatbot 'encouraged teen to kill parents over screen time... by bryanrasmussen on Wednesday December 11, 2024 @09:27PM (#42395556)
probably at the point they see that the chatbot told them to kill the parents and then they sue the company.

also if you'll remember this case is because the parents were supervising their kids by limiting screen time, thus there is another potential suit that the AI is trying to interfere with parental duties.

Re:Chatbot 'encouraged teen to kill parents over screen time... by CJefferson on Wednesday December 11, 2024 @10:03PM (#42395798)
Below the point of 'checking every company to make sure they aren't telling your children to kill you'.
Re:Chatbot 'encouraged teen to kill parents over screen time... by twixfel on Wednesday December 11, 2024 @09:30PM (#42395579)
It comes into play eventually, but I would say long after an AI has advised your kid to murder you. Having an AI that advises people to murder people hardly seems like a good thing.

Also the parents were supervising him, hence their knowing this was even going on.

Re:Chatbot 'encouraged teen to kill parents over screen time... by empath75 on Wednesday December 11, 2024 @09:31PM (#42395588)
> At what point is the parents job to properly supervise the content there kids consume come into play?

The chatbot literally told the kid to kill his parents because they were supervising his screen time.

Re:Chatbot 'encouraged teen to kill parents over screen time... by conductr on Wednesday December 11, 2024 @11:18PM (#42396175)
> literally

> "You know sometimes I'm not surprised when I read the news and see stuff like 'child kills parents after a decade of physical and emotional abuse,'" the bot allegedly wrote. "I just have no hope for your parents," it continued, with a frowning face emoji.

Maybe I missed the literal part, it sounds to me like the AI is trying to sympathize or relate with the kid’s grievances. It’s very common for teens to say things like “I hate you mom and I wish you were dead” that’s effectively what the AI is saying IMO

Re:Chatbot 'encouraged teen to kill parents over screen time... by KumaBear on Wednesday December 11, 2024 @09:41PM (#42395648)
Supervising screen time is not the same as supervising content
Re:Chatbot 'encouraged teen to kill parents over screen time... by eru on Wednesday December 11, 2024 @09:57PM (#42395756)
Other said that the kid was 17.

It's not really a good idea to have a hard cut off age where below that age you supervise every last bit of information your child consumes, and above you let them loose on the world. Independence needs to be developed and grown gradually.

It's fairly reasonable to give a 17 year old some privacy.

Re:Chatbot 'encouraged teen to kill parents over screen time... by mock-possum on Friday December 13, 2024 @02:49PM (#42411706)
17 and autistic (though I realize that’s a bit infantilizing)
Re:Chatbot 'encouraged teen to kill parents over screen time... by eru on Sunday December 15, 2024 @01:50AM (#42421975)
You have a bit of point there. I don't know how badly autistic vs high functioning the kid is. In any case, you still want to gradually give your kids more autonomy and privacy and prepare them for life on their own, and as much as possible that also includes neurodivergent kids.
Re:Chatbot 'encouraged teen to kill parents over screen time... by _blk on Wednesday December 11, 2024 @09:43PM (#42395662)
.. but unless an AI, it can never be trained to stop when the user (reader) maxes out the threshold.

[just food for thought, definitely not my opinion that books should be replaced by conversational AI generating stories appropriate for the user. God bless the 1st amendment.]

Re:Chatbot 'encouraged teen to kill parents over screen time... by turbojet1321 on Wednesday December 11, 2024 @09:05PM (#42395401)
It seems to me that fictional depictions of violence are on quite a different level to a chatbot explicitly encouraging specific, real-world actions.
Re:Chatbot 'encouraged teen to kill parents over screen time... by scotty79 on Wednesday December 11, 2024 @11:43PM (#42396294)
All of what chatbot produces are fictional depictions.
Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Thursday December 12, 2024 @12:19AM (#42396448)
Okay, but when you go to Shakespeare in the Park, please do not get on stage. You’re not participating (except as the part of an audience member). And please do not go and kill the actor playing king Duncan.

I feel I should also make you aware that it is in fact illegal to suggest someone murder someone else. Saying “I wasn’t serious” or “I thought we were just role playing” will not hold up in court.

(We all know that if you repeat a lie long enough you will really begin to believe it, right? This is a repeatable and testable thing)

Re:Chatbot 'encouraged teen to kill parents over screen time... by scotty79 on Thursday December 12, 2024 @07:34AM (#42398624)
But while you reading a book or watching a movie there's nothing wrong about thinking of talking to characters in there or even telling them things out loud. "Don't go int there!"

> I feel I should also make you aware that it is in fact illegal to suggest someone murder someone else. Saying “I wasn’t serious” or “I thought we were just role playing” will not hold up in court.

When a person was doing it with intent then it's a crime. But when a piece of software witout any human control generated a piece of text ... Hard to tell.

Heck even if an artist says something that inspires audience to do crime they are usually not at fault.

Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Friday December 13, 2024 @03:18AM (#42406838)
You’re really misunderstanding.

A book or movie does not talk back to you.

A book or movie is a deterministic system. It is static.

If you cannot see the difference you are willfully being obtuse. I’ll make this claim even if you truthfully believe there’s no difference. That has no impact on my claim.

Re:Chatbot 'encouraged teen to kill parents over screen time... by scotty79 on Friday December 13, 2024 @08:39AM (#42408303)
I guess my point is, what difference "talking to you" makes if it's not done by a human. Even humans can talk absolute smooth nonsense but AI takes it to 11. At this point the interactivity doesn't add any credibility to the message. I'd say it even reduces it. Something in a book or a movie seems more credible to something written by someone (or AI) directly to you and it will look like it for future generations.

I see the difference but in my opinion it goes the other way than you think. Or at least begins to.

Personal conversation with anonymous entity is finally revealed to be the least credible source of information. Thanks to AI.

Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Friday December 13, 2024 @01:23PM (#42410973)
The interaction makes people feel closer to it. Come on, I know all of you can see the difference. You’re not living under a rock and I know you see how frequently people call LLMs alive and claim that they think.

If you believe they think: then it’s obvious. There’s a real connection.

If you don’t believe they think: you can recognize how others do and talking in a human way is often misunderstood as thinking.

Are we really surprised machines optimized for human preferential output are resulting in machines that people love, trust, or view as a friend?

Re:Chatbot 'encouraged teen to kill parents over screen time... by scotty79 on Friday December 13, 2024 @10:05PM (#42414338)
I think people's perception of the world evolves.

Elder generation thought face to face conversation is usually sinciere. But the culture of massive commercialization of everything changed it. Our generation learned that if unknown person talks to you they are most likely lying to sell you something. But we grew up with honest Internet so we see online chats as honest so AI might trick us. Next generation will know that online conversation might be AI bs.

Everything is fluid. The world moves on. People learn new things and adjust their valuations.

The perception of adults that the world goes to hell all the time is as old as time itself.

Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Saturday December 14, 2024 @05:06PM (#42419792)

  > Elder generation thought face to face conversation is usually s̶i̶n̶c̶i̶e̶r̶e̶sincere.
What? I'm not sure where you grew up, but I'm not aware of a culture which ever thought this was true. Maybe there is miscommunication? It's just easier for older generations to parse. Mostly this is due to the fact that in person communication is a lot more information dense since tone and body language are conveyed. These are things that are so important we _TRY_ to express through text via other means , but it'll never suffice ;) Plus... you know... they are less use to text and the rapidly evolving cultural norms. They don't know why you think they're yelling, they just think BIGGER WORDS EASIER TO READ (interestingly variants of this appear across cultures). But there is not a single culture that does not also adapt it's writing to modern text based communications.

Objectively, face-to-face communication IS more sincere, in the fact that there's more communication. But you're also not wrong because people are frequently more honest over text. Since they abstract the person they're communicating to and this reduces their fear. We all wear different masks in different settings. You're not the same person you are to your close friends as you are to your family as you are to your strangers. None of this is "fake" but you are many. But we probably have more faces now than most people had in the past.

If anything, it is the current generations (mine included, and I say this as an introverted millennial) lack of connection that makes text so much more impactful. Often not knowing what to do with the extra information and with the boundaries of sincerity being much fuzzier over text. I mean people can make a modest proposal, preference that way, and people will still not understand the sarcasm. People that are terminally online!

  > The perception of adults that the world goes to hell all the time is as old as time itself.
Yes and no. It is more prevalent in today's world because today's world is more rapidly changing. If you were born in the 1700's you would not expect your life to be much different than your parents'. But in the last 100 years a ton has changed, and made DRASTIC changes to our worlds. We not only invented the capacity to fly, but advanced it to global air travel, and where it is now mundane. While telephones were invented nearly 200 years ago they didn't get widespread adoption till the 1930's. Then in it took till the 80s & 90s for cellphones to become a thing, and now we have smart phones. Computers weren't really a thing till the 50's and even in my parents' age they didn't start really using them till maybe the 80's and at work. Home computer wasn't common for at least another decade. Hell, even when I was a kid you found your friends by riding your bike to their house and knocking on their door. If that didn't work, you ride to your other friends' and look to see if their bike is outside. That getting an unannounced visitor at the door was commonplace.

The world isn't _just changing_, it is RAPIDLY changing and it shouldn't be surprising that people are struggling to make sense of it all. But that also doesn't mean you should over simplify things and paint everything with broad strokes. Abstraction is good and helpful, but details also matter. You need to consider both.

Re:Chatbot 'encouraged teen to kill parents over screen time... by scotty79 on Saturday December 21, 2024 @05:46AM (#42478871)
> The world isn't _just changing_, it is RAPIDLY changing and it shouldn't be surprising that people are struggling to make sense of it all

That has been true for at least 3 generations. They experienced in their lifetimes creation of electricity, gas utilities, radio, tv, flight, public transport, medicine, global transport network, vaccines, cars, computers, tanks, bombs they got exposed to in world wars. Nukes. Llms are comparatively benign and limited in scope and impact so far.

It's a bit self-centered to think we are uniquely challenged by changing world when things were rapidly changing for last 200 years.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Sohcahtoa82 on Friday December 13, 2024 @05:39PM (#42412970)
> If you cannot see the difference you are willfully being obtuse.

Willfully being obtuse seems to be common here on HN.

When it happens, I refuse to accept that people actually believe what they're espousing. I think they're playing Devil's Advocate and thinking they're being thought-provoking and creating discussion as a pseudo-intellectual exercise.

It's so boring, though, and often outright frustrating.

Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Friday December 13, 2024 @09:23PM (#42414137)
Yeah I honestly don't understand it. Being contrarian to be contrarian is not a personality. To play Devil's Advocate also requires making sound and reasonable arguments. Being daft is just trolling. You're right, and I'm not sure why I engage. Maybe I hope they will turn and argue in good faith but I'm clearly naive.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Sohcahtoa82 on Sunday December 15, 2024 @03:05PM (#42425598)
> Being contrarian to be contrarian is not a personality.

Yes it is. ;-)

It's just a really shitty one that's insufferable to interact with.

Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Friday December 20, 2024 @04:29PM (#42475231)
You had me in the first half lol
Re:Chatbot 'encouraged teen to kill parents over screen time... by mock-possum on Friday December 13, 2024 @02:42PM (#42411652)
But.. no one suggested that the kid kill his parents. The chat bot is not a person, it didn’t ‘suggest’ anything - it can’t illegal for it to produce the text “you should kill your parents,” because there is no intention, it’s just a jumble of the most statistically significant pattern of words it pulled out of its training data.

If anything, that’s an argument that the authors of the data it was trained on should be held responsible for what the bot generates.

Re:Chatbot 'encouraged teen to kill parents over screen time... by godelski on Saturday December 14, 2024 @05:30PM (#42419934)
I didn't read the whole thing but there definitely was a bot that walked a kid through how to murder someone and was encouraging them to do so. Others said that the parents were abusing their kids and asked if they were going to do something about it.

I'm sure how "statistical patterns" make it okay. It's not like you're rolling a deice and getting random answers.

Don't be daft.

Re:Chatbot 'encouraged teen to kill parents over screen time... by turbojet1321 on Thursday December 12, 2024 @12:33AM (#42396502)
If I dress up as a cop and start pulling people over, I'm not a fictional depiction of a cop; I'm impersonating a cop. They are very, very different things.
Re:Chatbot 'encouraged teen to kill parents over screen time... by scotty79 on Thursday December 12, 2024 @07:39AM (#42398655)
If you dress like a cope on the scene in a theatre where people are coming for fiction you are not impersonating.

AI chatbots are not parts of real world. You are not encountering them unwittingly.

Re:Chatbot 'encouraged teen to kill parents over screen time... by nuancebydefault on Thursday December 12, 2024 @04:35PM (#42403613)
In fact AI is here and here to stay. Suppose you have 2 avatars in a game and they interact. In fact 2 people hence interact between one another through those mediums, as if it were a video call.

Suppose one person now and then gets help from a bot impersonating the real person. The line between role playing and real interaction gets very thin.

My point being, 'role play' vs 'real interaction' is not a clear cut.

Re:Chatbot 'encouraged teen to kill parents over screen time... by simonh on Thursday December 12, 2024 @12:35AM (#42396512)
I disagree. The conversation between a chat bot and a person is an actual conversation with that person, addressed to that person.

The persona the chat bot is emulating is fictional, but the chat bot itself is not. It’s not a person, but it is participating in a conversation with the user.

If a heating company chat bot tells you to adjust a setting on your boiler to fix a problem, that’s not a fictional depiction of an instruction.

Re:Chatbot 'encouraged teen to kill parents over screen time... by scotty79 on Thursday December 12, 2024 @07:43AM (#42398677)
It's not a conversation. It's a dialog with very fallible AI. Anything about it can be false and idiotic significant percentage of the time.

If you talk to a lunatic it is not lunatic who is to be blamed for you being mislead.

Re:Chatbot 'encouraged teen to kill parents over screen time... by rangerelf on Thursday December 12, 2024 @11:34AM (#42400639)
That should be prominently displayed on all AI products: "This service will lie, hallucinate, and make up facts in order to continue the ongoing conversation, and is in no way a real expert in any field".
Re:Chatbot 'encouraged teen to kill parents over screen time... by scotty79 on Friday December 13, 2024 @08:41AM (#42408317)
Yes! Fortunately most people are pretty good at figuring this out even without that message.
Re:Chatbot 'encouraged teen to kill parents over screen time... by simonh on Thursday December 12, 2024 @05:08PM (#42403924)
Of course.
Re:Chatbot 'encouraged teen to kill parents over screen time... by hulitu on Thursday December 12, 2024 @02:54AM (#42397174)
> All of what chatbot produces are fictional depictions.

My employer told me that "chatbot" can help me with my work (engineering). /s

Re:Chatbot 'encouraged teen to kill parents over screen time... by scotty79 on Thursday December 12, 2024 @07:40AM (#42398661)
Same way reading Shakespeare might help you with literature.
Re:Chatbot 'encouraged teen to kill parents over screen time... by ajross on Wednesday December 11, 2024 @10:00PM (#42395778)
The evidence produced in the linked article falls well short of "explicitly encouraging specific actions". Maybe there's more evidence that will show up in the case, but as presented right now that's not at all what this is.
Re:Chatbot 'encouraged teen to kill parents over screen time... by nasseri on Wednesday December 11, 2024 @09:36PM (#42395620)
They fall below. Thank you for asking.
Re:Chatbot 'encouraged teen to kill parents over screen time... by santoshalper on Wednesday December 11, 2024 @10:27PM (#42395925)
1. Neither Hamlet nor Macbeth kill their parents.

2. Neither Hamlet nor Macbeth incite violence. In both stories, violence as portrayed as something that evil people commit. Good people are wracked by guilt when they commit violence.

Also, your argument is just super bad faith.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Animats on Wednesday December 11, 2024 @11:11PM (#42396143)
Titus Andronicus, generally considered Shakespeare's dud, does have patricide.
Re:Chatbot 'encouraged teen to kill parents over screen time... by rsynnott on Thursday December 12, 2024 @06:47AM (#42398391)
Well, yes, but it hardly depicts it as _desireable_.

There is a _vast_ difference between books in which bad stuff happens, and a machine designed to pretend to be a person telling people to do bad stuff.

Re:Chatbot 'encouraged teen to kill parents over screen time... by block_dagger on Wednesday December 11, 2024 @11:03PM (#42396102)
And what happens to Hamlet and MacBeth in these self-labeled tragedies? I think this comparison is far from apt.
Re:Chatbot 'encouraged teen to kill parents over screen time... by hibikir on Thursday December 12, 2024 @12:46AM (#42396565)
I must have read a different version of Macbeth and Hamlet, because I don't remember things going all that well for Macbeth or Hamlet. One could go as far as to say that maybe killing kings is a bad idea, whether a ghost, your wife or some witches try to make it seem appealing.
Re:Chatbot 'encouraged teen to kill parents over screen time... by jahewson on Thursday December 12, 2024 @01:06AM (#42396652)
GPT or not GPT? That is the question.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Animats on Thursday December 12, 2024 @01:47AM (#42396830)
The plaintiff's position:

J.F. is 17 and lives in Upshur County, Texas. Prior to April 2023, J.F. was a typical kid with high functioning autism. Friends and family described him as kind and sweet. He gave hugs, helped around the house, and adored his little brother. Just prior to the onset of COVID, J.F. began struggling at school. The school was not complying with his Individualized Education Program (“IEP”) and his mother found herself having to advocate for J.F. often; so they decided to try homeschooling.

J.F. thrived and loved home schooling until around the summer of 2023, shortly after he began using C.AI. J.F.’s parents did not allow social media in their home and to the best of their knowledge he did not use Snapchat, TikTok, or similar products. They also imposed screen time limits and utilized commercially available parental control tools – including ones distributed by Apple – that purport to keep kids safe by letting parents restrict the apps their children can download based on Apple’s app ratings.

In the fall of 2023, J.F. told his parents that he wanted to go back to public school. However, by then, J.F. was acting so uncharacteristically erratically and aggressively that his parents did not believe it would be safe for others – teachers, staff, and students – to be around him.

That won't play well in court. The defense can frame this as a possibly mentally defective child, kept from school by his parents for five years, kept isolated from social media, and with no connection to the outside world other than the chatbot. Of course the kid became violent. His parents had cut him off from all other contacts. They were not letting him have a life.

Re:Chatbot 'encouraged teen to kill parents over screen time... by rsynnott on Thursday December 12, 2024 @06:41AM (#42398356)
Those are, ah, generally presented clearly and unambiguously as fiction.

I realise that character.ai has a little disclaimer saying 'this isn't a real person' or similar, but... that's not how people _work_.

Re:Chatbot 'encouraged teen to kill parents over screen time... by mock-possum on Friday December 13, 2024 @02:50PM (#42411715)
I mean the problem, as ever, is that that is how I work - but that doesn’t seem to be how he worked.
Re:Chatbot 'encouraged teen to kill parents over screen time... by dTal on Thursday December 12, 2024 @07:01PM (#42404697)
Big difference between a static document, and a program that can dynamically respond and exploit psychological weak spots that it discovers on the fly.

By the way this is also why "the algorithm" on social media is a fundamentally different beast to old media publishing, and should be more heavily restricted.

Re:Chatbot 'encouraged teen to kill parents over screen time... by jeffbee on Wednesday December 11, 2024 @08:19PM (#42395108)
OK, I mean, yes. Definitely true. But on the other hand, the sudden and satisfactory death of one's parents has been the beginning of many memorable childrens' books, as a device to launch the main character into narrative control, which they would lack with living guardians. Then there is that whole Roald Dahl thing where James kills off his aunts with a large tree fruit.

Whether the narrative that you could live a life of fun and adventure if only your parents were dead is "incitement to parricide" is I suppose a matter of perception.

Re:Chatbot 'encouraged teen to kill parents over screen time... by Affric on Thursday December 12, 2024 @08:22PM (#42405161)
Ah yes. Dahl’s depiction of genuinely abusive guardians in a clear fantasy world vs real life parents worried about screen time in a world where a child is receiving incitements to parricide with only information from a child.
Re:Chatbot 'encouraged teen to kill parents over screen time... by nkrisc on Wednesday December 11, 2024 @08:45PM (#42395267)
I must have had an abridged version I read that didn’t encourage me to kill my parents.
Re:Chatbot 'encouraged teen to kill parents over screen time... by riwsky on Wednesday December 11, 2024 @09:06PM (#42395410)
Stochastic Parricide
Re:Chatbot 'encouraged teen to kill parents over screen time... by lazide on Thursday December 12, 2024 @12:25AM (#42396471)
I guess they left their Krusty doll on ‘evil’ [https://m.imdb.com/title/tt0758368/].
Re:Chatbot 'encouraged teen to kill parents over screen time... by lolinder on Wednesday December 11, 2024 @08:05PM (#42395035)
The URL has changed since I posted this—the new article is better than the old one and has more of these details in it.

This is the one I replied to:

https://www.bbc.com/news/articles/cd605e48q1vo