Yeah I can't imagine using the current model as part of an API (a popular use case for GPT-4) having seen this. I'm not sure it impacted their API edition of GPT-4 but this plainly shows how it could have given it leaked into another service in production, and that's bad enough.
I think GPT is fundamentally not good enough as an AI model. Another issue is hallucinations and how to resolve them, and an understanding of how information is stored in this black box and how to / if data can be extracted.
We have a long way to go and probably all these topics need to be answered first out of accuracy and even legal reasons. Up until then, GPT-4 should be treated as a convincing chat experiment. Don't base your startup or whatever on it. Use it as assistant where replies are provided in digestible and supervised fashion (NOT fed into another system) and you're an expert on the involved system itself and can easily see when it's wrong. Don't use GPT-4 to become an expert on something when you're a novice yourself.
I'm concerned about what happens when ChatGPT begins spewing coherent nonsense. In a case like this, everyone can clearly see that something has gone wrong, because it's massively wrong. What happens when thousands of "journalists" and other media people starts relying on ChatGPT and just parrots whatever it says, but what if what says is not obviously wrong?
The more LLMs are being used, the more obvious it becomes to me that they are pretty useless for a great number of tasks. Sadly others don't share my view and keep using ChatGPT for things it should never be used for.