News for hackers. Stuff that matters.
Re:OpenAI says over a million people talk to ChatGPT about s... by kace91 on Tuesday October 28, 2025 @04:31AM (#45730385)
>should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues.

Is that a debate worth having though?

If the tool is available universally it is hard to imagine any way to stop access without extreme privacy measures.

Blocklisting people would require public knowledge of their issues, and one risks the law enforcement effect, where people don’t seek help for fear that it ends up in their record.

Re:OpenAI says over a million people talk to ChatGPT about s... by probably_wrong on Tuesday October 28, 2025 @04:56AM (#45730557)
> Is that a debate worth having though?

Yes. Otherwise we're accepting "OpenAI wants to do this so we should quietly get out of the way".

If ChatGPT has "PhD-level intelligence" [1] then identifying people using ChatGPT for therapy should be straightforward, more so users with explicit suicidal intentions.

As for what to do, here's a simple suggestion: make it a three-strikes system. "We detected you're using ChatGPT for therapy - this is not allowed by our ToS as we're not capable of helping you. We kindly ask you to look for support within your community, as we may otherwise have to suspend your account. This chat will now stop."

[1] https://www.bbc.com/news/articles/cy5prvgw0r1o

Re:OpenAI says over a million people talk to ChatGPT about s... by kace91 on Tuesday October 28, 2025 @06:20AM (#45731073)
>Yes. Otherwise we're accepting "OpenAI wants to do this so we should quietly get out of the way".

I think it’s fair to demand that they label/warn about the intended usage, but policing it is distopic. Do car manufacturers immediately call the police when the speed limit is surpassed? Should phone manufacturers stop calls when the conversation deals with illegal topics?

I’d much rather regulation went the exact opposite way, seriously limiting the amount of analysis they can run over conversations, particularly when content is not deanonimised.

If there’s something we don’t want is OpenAI storing data about mental issues and potentially selling it to insurers for example. The fact that they could be doing this right now is IMO much more dangerous than tool misuse.

Re:OpenAI says over a million people talk to ChatGPT about s... by computably on Tuesday October 28, 2025 @05:09PM (#45739223)
Those analogies are too imprecise.

Cars do have AEB (auto emergency braking) systems, for example, and the NHTSA is requiring all new cars to include it by 2029. If there are clear risks, it's normal to expect basic guardrails.

> I’d much rather regulation went the exact opposite way, seriously limiting the amount of analysis they can run over conversations, particularly when content is not deanonimised.

> If there’s something we don’t want is OpenAI storing data about mental issues and potentially selling it to insurers for example. The fact that they could be doing this right now is IMO much more dangerous than tool misuse.

We can have both. If it is possible to have effective regulation preventing an LLM provider from storing or selling users' data, nothing would change if there were a ban on chatbots providing medical advice. OpenAI already has plenty of things it prohibits in its ToS.

Re:OpenAI says over a million people talk to ChatGPT about s... by kace91 on Wednesday October 29, 2025 @11:27AM (#45748111)
>Cars do have AEB (auto emergency braking) systems, for example, and the NHTSA is requiring all new cars to include it by 2029. If there are clear risks, it's normal to expect basic guardrails.

That's cool, but it is cool because it happens on device.

Analysing your conversations for certain patterns, and either cross referencing between conversations or keeping some sort of strike system, is the exact kind of feature that is a bad turn away from a dystopia.

Hopefully local llm's get good enough that we can avoid this issue altogether.

Re:OpenAI says over a million people talk to ChatGPT about s... by arowthway on Tuesday October 28, 2025 @05:19AM (#45730706)
Are people using ChatGPT for therapy more vulnerable than people using it for medical or legal advice? From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy.
Re:OpenAI says over a million people talk to ChatGPT about s... by latexr on Tuesday October 28, 2025 @05:37AM (#45730824)
> Are people using ChatGPT for therapy more vulnerable than people using it for medical or legal advice?

Probably. If you are in therapy because you’re feeling mentally unstable, by definition you’re not as capable of separating bad advice from good.

But your question is a false dichotomy, anyway. You shouldn’t be asking ChatGPT for either type of advice. Unless you enjoy giving yourself psychiatric disorders.

https://archive.ph/2025.08.08-145022/https://www.404media.co...

> From my experience talking about your problems to the unaccountable bullshit machine is not very different then the "real" therapy.

From the experience of the people (and their families) who used the machine and killed themselves, the difference is massive.

Re:OpenAI says over a million people talk to ChatGPT about s... by terminalshort on Tuesday October 28, 2025 @11:11AM (#45733889)
I've been talking about my health problems to unaccountable bullshit machines my whole life and nobody ever seemed to think it was a problem. I talked to about a dozen useless bullshit machines before I found one that could diagnose me with narcolepsy. Years later out of curiosity I asked ChatGPT and it nailed the diagnosis.
Re:OpenAI says over a million people talk to ChatGPT about s... by danaris on Tuesday October 28, 2025 @07:43AM (#45731573)
Then...

Maybe the tool should not be available universally.

Maybe it should not be available to anyone.

If it cannot be used safely by a vulnerable class of people, and identifying that class of person sufficiently to block use by them, and its primary purpose is simply to bring OpenAI more profit, then maybe the world is better off without it being publicly available.

Re:OpenAI says over a million people talk to ChatGPT about s... by kace91 on Tuesday October 28, 2025 @08:05AM (#45731716)
>If it cannot be used safely by a vulnerable class of people, and identifying that class of person sufficiently to block use by them

Should we stop selling kitchen knives, packs of cards or beer as well?

This is not a new problem in society.

>and its primary purpose is simply to bring OpenAI more profit

This is true for any product, unless you mean that it has no other purpose, which is trivially contradicted by the amount of people who decide to pay for it.

Re:OpenAI says over a million people talk to ChatGPT about s... by danaris on Tuesday October 28, 2025 @08:33AM (#45731993)
There's a qualitative difference between knives and publicly-available LLMs.

Knives aren't out there actively telling you "use me, slit those wrists, then it'll all be over".

Re:OpenAI says over a million people talk to ChatGPT about s... by kace91 on Tuesday October 28, 2025 @09:15AM (#45732474)
I don’t disagree that they are clearly unhealthy for people who aren’t mentally well, I just differ on where the role of limiting access lies.

I think it’s up to the legal tutor or medical professionals to check that, and providers should at most be asked to comply with state restrictions, the same way addicts can figure on a list to ban access to a casino.

The alternative places openAI and others in the role of surveilling the population and deciding what’s acceptable, which IMO has been the big fuckup of social media regulation.

I do think there is an argument for how LLMs expose interaction - the friendliness that mimics human interaction should be changed for something less parasocial-friendly. More interactive Wikipedia and less intimate relationship.

Then again, the human-like behavior reinforces the fact that it’s faulty knowledge, and speaking in an authoritative manner might be more harmful during regular use.