I agree with everything you're saying, but there are no legal protections for incitements to violence, or other problematic communications (such as libel) by an LLM. It may be that they provide a very valuable service (though I don't see that), but the risk of them crafting problematic messages is too high to be economically viable (which is how it seems to me).
As it stands, this LLM seems analogous to a low-cost, remote children’s entertainer which acts as a foolish enabler of children’s impulses.
But yes, maybe a service such as this can't exist in our legal framework. Which on the internet likely just means that someone will launch a more shady version in a more favorable jurisdiction. Of course that shouldn't preclude us from shutting down this version if it turns out to be too harmful. But if the demand is there, finding a legal pathway to a responsibly managed version would be preferable (not that this one is perfectly managed by any means)
Not offer the product? Stop the chat when it goes off the rails?
You're taking it as a given that Character AI should exist. It is not a person, but an offering of a company made up of people. Its founders could have started a different business altogether, for example. Not all ideas are worth persuing, and some are downright harmful.
Given the technology we have, I'm not entirely sure what Character AI could have done differently here. Granted, they could build in more safeguards, and adjust the models a bit. But their entire selling point are chat bots that play a pre-agreed persona. A too sanitized version that constantly breaks character would ruin that. And LLMs are the only way to deliver the product, unless you dial it back to one or two hand-crafted characters instead of the wide range of available characters that give the service its name. I'm not sure they can change the service to a point where this complaint would be satisfied.