For ethics, how is this more dangerous than trials for a new heart medication that might kill patients? In fact, the only way to know for sure, is to try and see.
In terms of early results, these kinds of toy projects will be the frontline since any lawyer would tell you to not touch this with a 10ft pole. But I doubt a Github project would face that scrutiny since it takes technical skills to setup which requires knowledge and intent on the part of the user.
Clinical trials happen through an established regulatory framework to minimize harm to participants, for starters…
Knowing that hundreds of people end their lives /after/ seeking care in our current mental health system, I’m not afraid of exploring other options.
I do believe, however, that developers in this space must not over-represent the capability of their systems and have a duty to strongly warn users that the system can act unpredictably and give harmful advice. Resources to contact health professionals and warnings to contact 911 if there is a life-threatening emergency are also obligatory.
With those disclosures, I don’t see how a self-directed discussion with a language model could be overly harmful.
This has the evidentiary burden entirely backwards. We have evidence for the efficacy of various forms of psychological and psychiatric interventions and none for that produced by LLMs. I’m not even sure if it’s possible or ethical to test them for this purpose, as it would require OpenAI levels of RLHF training to get them to adhere to current guidelines and standards of care.