Imagine how much pressure must exist within Microsoft for that to happen.

Back in 2016, Microsoft took the chatbot Tay down in 1-2 days (https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...). This time, some team made the calculation that it's better to learn on end users, even with the press hit, than to learn on paid testers (like with more RLHF). And that's despite being behind a waiting list; Bing Chat's waiting list would grow either way.

I'm not an AI risk zealot or even expert, but next time you hear "AI risk," think "Humans deciding to release products that they know have major user-facing flaws."

Almost nobody is going to knowingly release something that will cause major real-world damage, but many will intentionally release things that they know have gaping flaws. The effective AI risk margin isn't "We thought <new product> was great, so it was impossible to imagine that it could cause real-world damage," it's "We thought <new product> only had major user-facing flaws, not that it would cause real-world damage."