Another factor I think that plays in here is a bit of hubris about who has access to truth. Someone of Freeman Dyson's caliber is used to being the one who can see further into complexity to find truth. When someone has these talents, they tend to see all problems as problems they have special access to. If they see a lot of people without their abilities believing a thing with a lot of complexity, this creates the conditions to flip to being contrarians about that subject. I believe this kind of reactive contrarianism has happened in climate science quite a bit.

The bias here is that highly talented people sometimes cannot fathom that the ability to see some truths is broadly distributed. But in the case of human behavior, our ability to see truth is broadly distributed, much more so than the science someone like Freeman Dyson can do. And if that human behavior factor is the key variable, then the broad consensus can be correct.