As you say, often you check up on the LLM's "reasoning" and it doesn't follow at all, or you can easily get it to contradict itself with just as much certainty as it had about its previous convictions.

It is very scary to me that people are entrusting potentially life-altering decisions to these things.