Just writing something bad doesn't actually mean something bad happened.
These days it seems like people are oversensitive to how things are said to them, and what things are said to them.
Rohingya is a textbook example of blind optimisation and lack of context awareness. FB looked at a region, and people communicating in language they didn't understand. But they did see that certain symbols and/or combinations of symbols got a lot of engagement. If you're after money, you want to amplify the use of those symbols and hopefully generate lots more similar content.
Turns out that's a morally reprehensible thing when the people using those symbols were advocating genocide. (It was good for the revenue while it lasted, though.)
With LLMs and their hardcoded guard rails, I suspect we're going to see the danger emerge from the other side. Instead of actively spewing hatred, they will be used for mass sock-puppetry and opinion amplification on a massive scale. Think simple sabotage field manual for 21st century, but weaponised thousand-fold.
What matters is the reader not the writer.
Facebook was accused of making it too easy for people to communicate. And people felt Facebook should police what people say to each other. I don't agree, but even if I did, that's not the same thing as what we are discussing.
Is that bad? It's just a language model - it says things. Humans have been saying all sorts of terrible things for ages, and we are still here.
I mean it makes for nice headline "model said something racist", but does it actually change anything?
These aren't decision making AI's (which would need to be much more careful), they are language models.