Higher probably responses are not necessarily “truthful” or “logical” in any given context.

Here* you can see an example of what happens when you always choose the highest probability.

[*] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...