Perhaps there is a model of the world that is 90% compatible with the most usual phrases used in the training dataset. That model could be 10% of the time absolutely wrong about the real state of our world. Since the way in which LLM construct a model of the world is not ruled based, it is difficult to design an adversarial attack to detect the pitfalls of the model.

We know that LLMs are trained just to predict the next token and that the model of the world that we assume she has is based on that. This intuitively suggests that such an adversarial attack could be devised and exploited, but at the same time it indicates that such attack is very difficult to design.

Because LLMs model are weak at math problems and at logic reasoning we could look for those fields that reflect the weakness of LLMs, so we should base the attack on concepts that are an expansion of those known weaknesses.

I haven't found any deep thought about probabilistic analysis and hypothesis of how the weakness of LLMs are multiplied when applied to more elaborated concepts, but we all know that the problem of logic is that one weak or false argument break the strength of a chain of logical reasoning, to summarize: we should look for those strands that when loosed break concepts apart.