LLMs are really good at repeating text that they've just seen. Very occasionally they'll mix up a word or two, but it's not at all challenging for them to regurgitate text from a previous section of input.

I have yet to see a documented example of a system prompt leak that was NOT the real system prompt. Have you seen one?

> I have yet to see a documented example of a system prompt leak that was NOT the real system prompt. Have you seen one?

I have yet to see a convincing proof that there is a teapot in an orbit around the sun between earth and mars.

The people who can debunk these system prompts claims have zero incentive to do so.

The fact that a lot of these "leaks" happen yet non of the prompts are the same or even seem to be different iterations of the same family of system prompts, should tell you all you need to know.

Show me same examples of a leaked system prompt that came out different ever time and I'll concede that system prompt leaks are likely junk.

My experience has been the opposite: the way you confirm a system prompts leaks leak is to try it several times and check that you get the same result, and I've seen that done many times over the past few years.

yeah, just assumed it is.

The chatbot craze in microcosm.

I think it's worth elaborating.

Loosely, LLMs give plausible responses. And LLMs are really good at writing confident-sounding responses.

LLM output is as if someone is replying with the sole purpose of appearing helpful and knowledgeable.

I wouldn't trust opinions on LLMs from people who are entirely positive or entirely negative: the technology is just too mixed for that. I'd say it's useful for someone to have had a bad experience with LLMs (e.g. LLMs being confidently wrong), as well as making use of LLMs for things they're powerful at. (e.g. "small" programming tasks).