Information-theoretical arguments are powerful, but they're not the only worthwhile approaches. You can't use an information-theoretic argument to teach someone information theory, and they'll find it easier to grok the consequences of information theory if they have other concepts to relate it to. Having multiple different routes to a given understanding is useful.

Wittgenstein was studying the nature of language, something closer to mathematics than to physics. And he came up with these ideas no later than 1933: Shannon only published his work on information theory in 1948. That Wittgenstein's later work was validated by advances in science over a decade later suggests that "philosophical mumbo jumbo" does not characterise it well. Indeed, perhaps there's something to learn from it.