That is unfortunately true. It seems to me, at some level, the whole problem is less about knowledge than about which sources someone regards as trustworthy and which not.

We can only experience very little of the world directly. For most of our knowledge, we have to rely on intermediaries which tell us about the world. If some of those intermediaries offer contradictory information or offer information that contradict with your own experiences or discredit other intermediaries then things get problematic and you have to pick side about which intermediaries to trust.

I think psychologically, this is a hard problem to solve.