It is important to understand the limitations of logic and maths. They are fundamentally tautological systems - they can’t give you genuinely new knowledge about the world. It can’t tell you if purple swans exist or whether planetary orbits are circular or elliptical. You need empirical evidence for that.

So information theory might give you a measure of the complexity of a theory, but can’t (on its own) say anyting about whether it is true or not.