News for hackers. Stuff that matters.
Re:Chatbot 'encouraged teen to kill parents over screen time... by joe_the_user on Wednesday December 11, 2024 @06:19PM (#42394192)
If you actually page through the complaint, you will see the chat rather systematically trying to convince the kid of things, roughly "No phone time, that's awful. I'm not surprised when I read of kids killing parents after decades of abuse..."

I think people are confused by this situation. Our society has restrictions on what you can do to kids. Even if they nominally give consent, they can't actually give consent. Those protections basically don't apply to kids insulting each other on the playground but they apply strongly to adults wandering onto the playground and trying to get kids to do violent things. And I would hope they apply doubly to adult constructing machines that they should know will attempt to get kids to do violent things. And the machine was definitely trying to do that if you look at the complaint linked by the gp (and the people who are lying about here are kind of jaw-dropping).

And I'm not a coddle the kids person. Kids should know all the violent stuff in the world. They should be able to discover it but mere discovery definitely not what's happening in the screenshots I've seen.

Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @07:38PM (#42394845)
This is cherry picked content to play out a story for the case. They picked 5 worst samples they could think of in the worst order possible probably out of 1000+ messages.

The root cause here is the parents. It's visible behind those screenshots. The child clearly didn't trust their parents and didn't feel they cared, listened or tried to understand the problems the teen was going through. This is clearly a parenting failure.

There's far worse content out there than these mild messages that teenagers will be in contact with, starting with 4chan, to various competitive video games to all other sorts of weird things.

This is a cop out for parenting failure, where parents are looking to play victims, since they can't take responsibility for their failures and AI seems like something that could make them feel better.

At 17, it's humiliating to have phone taken away from you in such manner, then your parents going through your phone to find those text messages in the first place. Then making a lawsuit out of this portraying all the intimate details to the public. These parents seem to have 0 care for their child.

Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Wednesday December 11, 2024 @08:09PM (#42395058)
> they picked 5 worst samples they could think of in the worst order possible probably out of 1000+ messages

0.5% is a really high fraction for fucking up to the point of encouraging kids to murder!

Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @08:12PM (#42395069)
There was no encouragement of murder. Paraphrased, the AI said that given controlling nature of some parents it's no surprise that there are news articles of "children killing their parents". This is not an encouragement. It is a validation of how the kid felt, but in no way does it encourage to actually kill their parents. It's basic literacy to understand that it's not that. It's an empathetic statement. The kid felt that parents were overly controlling, AI validated that, role playing as another edgy teenager. But not actually suggesting or encouraging it.
Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Wednesday December 11, 2024 @08:29PM (#42395173)
> the AI said that given controlling nature of some parents it's no surprise that there are news articles of "children killing their parents"

Now put that in a kid’s show script and re-evaluate.

> It's basic literacy to understand that it's not that

You know who needs to be taught basic literacy? Kids!

And look, I’m not saying no kid can handle this. Plenty of parents introduce their kids to drink and adult conversation earlier than is the norm. But we put up guardrails to ensure it doesn’t happen accidentally and get angry at people who fuck with those lines.

Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @08:33PM (#42395197)
It's crazy to me the sentiment here and how little respect there is to an intelligence of 17 year olds that they are unable to understand that it's not actually an encouragement to kill someone. It's same or worse vibes as "video games will make the kids violent".
Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Wednesday December 11, 2024 @11:22PM (#42396195)
> same or worse vibes as "video games will make the kids violent"

We have no evidence of video games causing violence. We have evidence of kids killing themselves after talking to bad chatbots.

Re:Chatbot 'encouraged teen to kill parents over screen time... by ToValueFunfetti on Thursday December 12, 2024 @12:05AM (#42396388)
We must have magnitudes more evidence of kids committing violence after playing violent video games; video games are much more popular and have been around a lot longer, and juvenile violence is more common than suicide.
Re:Chatbot 'encouraged teen to kill parents over screen time... by Kbelicius on Thursday December 12, 2024 @12:56AM (#42396611)
> more evidence of kids committing violence after playing violent video games

GP said: "no evidence of video games causing violence.", which is completely different to what you wrote. I'm sure a lot of violence is committed after lunch.

Re:Chatbot 'encouraged teen to kill parents over screen time... by tmtvl on Thursday December 12, 2024 @04:45AM (#42397769)
Yes, but GGP also said that kids commited suicide after talking to a chatbot. I agree that there's no evidence for video games causing violence (rather the opposite), but this double standard that GGP is setting deserves calling out.
Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Thursday December 12, 2024 @08:00AM (#42398779)
> GGP also said that kids commited suicide after talking to a chatbot

A chatbot telling a kid to kill themselves and then the kid killing themselves seems like causation [1].

[1] https://www.nytimes.com/2024/10/23/technology/characterai-la...

Re:Chatbot 'encouraged teen to kill parents over screen time... by ToValueFunfetti on Thursday December 12, 2024 @08:12AM (#42398840)
Sure, but so does a video game telling a kid to commit violent acts and then the kid committing violent acts. I don't think video games cause violence and I'm open to the possibility that chatbots cause suicide, but if we're going to compare evidence for each, we shouldn't do it in a biased way.
Re:Chatbot 'encouraged teen to kill parents over screen time... by vacuity on Thursday December 12, 2024 @09:45AM (#42399537)
It's prima facie more plausible for chatbots to cause suicide, considering that chatbots are more personal and interactive than even video games. There's a distinct difference, I would think, between what is obviously fake murder in a fake setting and being sympathized with, like one human to another, on thinking about actual murder. And while chatbots explicitly have the warning that they are not real people, I would not expect a person with an underdeveloped prefrontal cortex and possibly pre-existing mental health troubles (again, this can apply to video games too, but, I imagine, to a lesser degree) to fully act accordingly.
Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Thursday December 12, 2024 @10:03AM (#42399708)
> so does a video game telling a kid to commit violent acts and then the kid committing violent acts

Big difference is the video game industry had studies to back them up. Where are the data for chatbots? On the benefits? Lack of risk? Is there a single child psychologist in the ranks of these companies?

Video games are also rated, to help parents make age-appropriate decisions. Is Character.ai age gated to any degree?

Re:Chatbot 'encouraged teen to kill parents over screen time... by bumby on Friday December 13, 2024 @09:15AM (#42408567)
Is it at least fair to say the data is mixed? Not my field, but there is some research to suggest video games may increase short-term aggression and desensitization to violence.

And while industry research doesn’t equate to bad research, it should be held to a higher standard simply because of the obvious incentives. Would you automatically accept tobacco company research to make strong conclusions about the safety of smoking?

Re:Chatbot 'encouraged teen to kill parents over screen time... by bumby on Friday December 13, 2024 @09:26AM (#42408671)
Tbf, strict causality is very difficult to prove in social sciences, no? Meaning, most of the studies for/against the link between video games and violence can't meet that threshold. Social science isn't physics and I don't think it's fair to treat them the same.
Re:Chatbot 'encouraged teen to kill parents over screen time... by numbsafari on Wednesday December 11, 2024 @10:01PM (#42395783)
There’s a reason why the Supreme Court held that it is cruel and unusual punishment to incarcerate a minor for life without parole.
Re:Chatbot 'encouraged teen to kill parents over screen time... by empath75 on Wednesday December 11, 2024 @09:32PM (#42395596)
The kid is autistic. There are younger kids than 17 year olds using that app.
Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @09:56PM (#42395752)
It's a whole conversation with context being an edgy teenager conversating with another edgy LLM teenager. I don't know if you've ever been teenager, but despite this being long time ago for me, I still feel like I can relate to that mindset, and it seems clear to me that the LLM is just going along with this edgy teenager vibe. If the other participant is as such and if the prompt is as such, this will yield in a result like this. I'm borderline autistic, and had many social issues as a teenager, and I absolutely loved any sort of dark humor as well at those ages. Well, I still do love dark humor, but I did back then too. Him being "autistic" here is just used for the court case. It's clear he's high functioning and has enough intelligence to understand what is wrong and what is right.
Re:Chatbot 'encouraged teen to kill parents over screen time... by twixfel on Wednesday December 11, 2024 @09:36PM (#42395619)
not all 17 year olds are equally intelligent you know? And if even one kid is convinced to murder his parents by an AI then that’s one too many.
Re:Chatbot 'encouraged teen to kill parents over screen time... by mewpmewp2 on Wednesday December 11, 2024 @09:57PM (#42395759)
Based on just the screenshots and material in the court case, it occurs to me in this case the kid seems more intelligent to me than his parents though. And I'm not even joking or being facetious. Kid is fact checking what AI is telling about bible, etc, etc, being skeptical about religion despite the bring up, etc. It's just small example, but it's otherwise how he writes as well.

The LLM in terms of edginess is just going to build on your own edginess assuming it is uncensored. It is not going to convince you of something out of nowhere.

Given clear hint that someone is happy with dark humour, LLM should be able to throw some of it out back.

I am just sad the kid has this type of gaslighting parents making him feel that he is in the wrong when he seems more intelligent than they are.

Re:Chatbot 'encouraged teen to kill parents over screen time... by rsynnott on Thursday December 12, 2024 @06:53AM (#42398416)
Ah, the Henry II defence. "Well, _technically_, I didn't tell them to kill the priest."

Context is everything; if this had been a conversation with a human it would be hard not to read it as malicious.

Re:Chatbot 'encouraged teen to kill parents over screen time... by gavindean90 on Wednesday December 11, 2024 @08:28PM (#42395167)
Yea I’d like at least 5-6 9s in this metric
Re:Chatbot 'encouraged teen to kill parents over screen time... by rsynnott on Thursday December 12, 2024 @06:51AM (#42398409)
This is, really, yet another example of the trouble with the "LLMs are correct 90% of the time, and only go totally off the rails 5% of the time" marketing line. There are remarkably few use cases, it turns out, where that is okay; you really need it to _not matter at all_ if the output is arbitrarily wrong.

(I suspect character.ai was originally conceived precisely because it appeared to be a usecase where LLM unreliability would be okay, the creators not having thought sufficiently carefully about it.)

Re:Chatbot 'encouraged teen to kill parents over screen time... by JumpCrisscross on Thursday December 12, 2024 @08:01AM (#42398787)
> remarkably few use cases, it turns out, where that is okay; you really need it to _not matter at all_ if the output is arbitrarily wrong

It's probably true in most cases. Those cases don't cover kids.

Re:Chatbot 'encouraged teen to kill parents over screen time... by rsynnott on Thursday December 12, 2024 @08:44AM (#42399065)
I think there are a lot of cases where it _seems_ to be true, until you think through the details. Most cases where it actually _is_ true are, in practice, very low impact; the big proven one seems to be, essentially, generation of high-volume spam content.
Re:Chatbot 'encouraged teen to kill parents over screen time... by NiloCK on Wednesday December 11, 2024 @08:26PM (#42395154)
Your honor, this entire case is cherry picked. There are thousands of days, somehow omitted from the prosecution's dossier, where my client committed ZERO murders.