Funny thing: the raw GPT4 is incredibly well calibrated. If it gives you an answer with a 40% probability of being correct - it knows that it's 40%.

After reinforcement learning on the other hand it copied the human fallacy: it's very sure when it's very sure, but after that it's stuck at "yeah, it could be true".