The chatbot in that article did have safeguards in place to stop it from listing suicide methods, but he just kept asking until he found a jailbreak. He was looking for someone to give him the answer he wanted all along

Past a certain point is there anything that will stop someone that determined to find something to validate their own self-destructive viewpoint? If not AI then a site or faction or person on Telegram with a pro-suicide opinion could have the same effect, and it would still be just as tragic but no longer be noteworthy. It didn't sound like he had a happy life to begin with with the AI ruining it completely, like it sounds

I had an LLM tell me to that killing myself makes sense when recounting a low point in my life to it and how I'd contemplated suicide. I wasn't looking for any kind of "jailbreak" in prompts, this was following a genius on HN saying it's a "great alternative to a therapist". No, it isn't fit for purpose.

> It didn't sound like he had a happy life to begin with

Which are exactly the kind of people who are going to be using LLMs as a "therapist". See the problem?

LLMs make mistakes quite often. It is unreasonable to expect that it stops making mistakes just because suicide is mentioned.

LLM is a tool. There are many tasks it is good at and even more tasks where it sucks e.g., GPT-4 can fail on trivial chess questions.

Avoid permanent solutions to temporary problems.

He had mental issues before that - the bot simply reinforced his decisions me thinks.

Yes, which is the point: if you substitute a human therapist for an LLM, this is what's going to happen because "guard rails" can't account for every scenario. If half a billion dollars doesn't buy a bug-free game, why would it buy a safe LLM?

Does suicide encouraged by a therapist happen with human therapists too? Probably, but likely much less common as a suicide is going to hurt your reputation.

Millions upon millions of people have mental health issues. Chatbots that reinforce those issues shouldn't be dismissed with a "simply... me thinks".