Syntactic validity is not semantic validity.
Word predictors not world state predictors
Text prediction not fact prediction
Frankly though the best answers are
1) let’s talk to infosec first
2) hey what’s the error rate ?
I have noticed that GPT etc had a big "wow effect" on me, the first impression can be great because it's simply not a level of language one would expect from a computer. But prod it long enough with prompts and somehow a pattern emerges: the output never contains a higher amount of information than the prompt. Copilot can type out pages of boilerplate code because boilerplate code is noise, it can write a quicksort because the word "quicksort" already contains all the information necessary to define its behaviour.
You can only make out when you actually push the blasted thing.
It is text gen. Just examine the premise of chain of thoughts.
Chain of thoughts promoting shouldn’t make a difference to a world model. Definitely not to a model that logic has emerged out of.
It makes a difference to a decompression function.
Edit; you may like this: https://hai.stanford.edu/news/ais-ostensible-emergent-abilit...
Like, "production tools are semantic time bombs" because at first they are boring, and then the timer on the bomb runs out, and they - like "proof of concept LLM tools - blow your mind.
Tick tick BOOM.
This will be my new favorite quote for whenever someone tries to pitch his latest LLM idea