News for hackers. Stuff that matters.

A comment on Better Intuition for Information Theory.

Re:Better Intuition for Information Theory by knzhou on Tuesday December 03, 2019 @01:57PM (#21695378)
This is nice, but I could've sworn I saw the exact same diagrams in all the old-school information theory books I originally learned it from. I've drawn the I-diagrams the author gives in my notes quite a few times. I'm guessing the point of the paper is that it goes further than that?
[ Parent | on HN ]
Re:Better Intuition for Information Theory by kthielen on Tuesday December 03, 2019 @02:53PM (#21696006)
The paper is from 1991, when was your textbook written?
[ Parent | on HN ]
Re:Better Intuition for Information Theory by mturmon on Tuesday December 03, 2019 @03:05PM (#21696138)
The first place I saw these diagrams is in the first edition of Tom Cover and Joy Thomas's excellent textbook - copyright 1991.

But the Venn diagrams are presented in their book as mnemonic tools, and not in terms of entropies of an underlying "I-measure" as in the article. (Of course, the book is an introduction, not in a theorem/proof format.) So you're not sure how far the notions can be extended.

The world of information theory was small then, and the author of the paper would have been well-known to both Cover and Thomas.

[ Parent | on HN ]
Re:Better Intuition for Information Theory by knzhou on Tuesday December 03, 2019 @03:27PM (#21696336)
Ah, I see. I was under the impression the paper was new.
[ Parent | on HN ]