News for hackers. Stuff that matters.

A comment on Cross Entropy. Parent comment

Re:Cross Entropy by tziki on Wednesday May 23, 2018 @03:50PM (#17137560)
> The entropy is the expected number of bits when this allocation is done using the incorrect distribution

Is there any source that would derive and/or explain this more in-depth? I've been trying to develop an intuition for this, but haven't come across a good explanation.

[ Parent | on HN ]
Re:Cross Entropy by banachtarski on Wednesday May 23, 2018 @04:20PM (#17137839)
The other reply mentioning "kullback-leibler divergence" (aka KL divergence) is what you need to understand as this is the fundamental concept. Minimizing this quantity is equivalent to minimizing the given "cross-entropy loss" expression. More generally to understand where this comes from, you'll want to read about information theory.
[ Parent | on HN ]
Re:Cross Entropy by banachtarski on Wednesday May 23, 2018 @05:07PM (#17138303)
https://rdipietro.github.io/friendly-intro-to-cross-entropy-...

I found this which seems much better

[ Parent | on HN ]
Re:Cross Entropy by pizza on Wednesday May 23, 2018 @03:54PM (#17137600)
the OG http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...

also lookup kullback-leibler divergence

[ Parent | on HN ]