The most interesting part is that there is no direct line between someone's accidental death and a chatbot giving life-threatening advice.

Imagine one of the models that has "accidental-deaths-via-bad-advice" just slightly turned up, with the model-provider's intent being to kill 5% more people per year.

Killing your customers is not the best way to stay in business

If you're paranoid (or a hawk), imagine a Chinese LLM that only offers fatal advice when queried English, or an American LLM that only offers fatal advice when queried in Chinese. Or American and Russian models which only offer up fatal advice when queried in German, Finish, or Danish.

(Exceptions like euthanasia clinics apply.)

Wonder how many AI deaths have occured that we dont know about(since they presumably died). With the adoption numbers we are seeing it much have happened already.

I'd be surprised if it was less than hundreds, or more than hundreds of thousands.

High hundreds of thousands feels like the upper limit before it would show up in statistically noticeable changes in patterns of deaths in some demographic.

High hundreds of individuals would still be "one in a million fatal errors over a few years", which seems better than I'd expect given I've personally had ChatGPT tell me that Solanum nigrum berries were "black tomatoes" (they're not usually fatal, but are a bit toxic, and no I did not eat them).