One of our biases is towards conservation of mental energy. If we can, we choose the easiest path. Which means that we actively prefer experts who are smart, well-informed, and certain of their beliefs. Because we can then be comfortable handing our thinking over to them. Why bother questioning the expert when we'd surely agree if we just knew enough? And then confirmation bias keeps us from ever questioning their results.
The result, as https://www.amazon.com/Expert-Political-Judgment-Good-Know/d... documents, is that our most popular and highest paid pundits have those characteristics. But those experts are ALSO the ones who are worst at making predictions about the world. As I pointed out in https://www.lesswrong.com/posts/TMgfapzbt5qp4Hszf/doubt-cert..., we should really be concluding "likely has cognitive bias" instead of "the expert says it, so case closed".
The documented result is that we wind up trusting the experts that we should trust least!
Trust A over B when C (subject to real time limits on computing C) = ok
Trust Less = harmful
As you note, we need to decide who to trust, and we can't do sufficient research. So we need heuristics.
The heuristics that I recommend are these:
1. Certainty, especially if there is debate among experts, indicates likely cognitive bias.
2. The ability to discuss how multiple perspectives would lead different people to different conclusions, suggests good reasoning.
3. A documented history of quantitative reasoning, along with data showing historical accuracy, is the best evidence of a trustworthy source.
I base those heuristics on the research in https://www.amazon.com/Expert-Political-Judgment-Good-Know/d.... Applying this heuristic to "sounds certain" results in a conclusion, that is an invitation to further research that may or may not happen.
To what degree did you take the degree of rationality you've achieved into consideration (Rationalists aspire to rationality, though they often forget that) during your analysis?
Who does? Not everybody, everywhere, certainly.
Holding fixed abstract ideas in mind for long periods of time, with conviction and a drive to keep them all consistent is a very peculiar, intellectual way of seeing the world, mostly only useful for debate team members, monks, academics, and political pundits. Especially for the countless political and metaphysical ideas that have no relevance to daily actions.
Most people don’t need to do that, and many who do it now have learned to perform it because it’s come to seem normal or even celebrated. But even a lot of them are terrible at it, hanging onto all kinds of sloppy beliefs and inconsistencies.
By default, most people just do shit and say shit and carry on with their day. Their natural BS detector looks like “God, the news program is awful. Turn it off.” and their natural political orientation is whatever makes for the most satisfying conversation with their neighbors.
What you might want to do, instead of all the complexity you’re trying to navigate and systematize, is to just unplug a bit and busy your hands instead of your brain.
One group cannot speak on all things, they just cannot have all the required knowledge, experience, and practice to accurately judge more than a few corners of reality. The modern jack of all trades can only really have a surface level understanding of most things.
You can only accept information from others as "someone wanted to communicate this to me" and speculate on their real reasons for doing so.
We just have to do the best we can with the limited, imperfect information available to us. Our very senses lie to us.
If your notion of rationality requires there to be Perfect Truth then you're SOL right at the start.