News for hackers. Stuff that matters.
Re:A Friendly Introduction to Cross-Entropy Loss (2016) by happy4crazy on Thursday June 07, 2018 @09:19AM (#17255485)
I personally don't find the "bits"y explanation of entropy/cross-entropy/KL etc. to be all that intuitive; as fundamental as it may be, I don't think about compression/encodings all that often. I've always preferred the "surprise" interpretation: http://charlesfrye.github.io/stats/2016/03/29/info-theory-su...

In short: given some event of probability p, -log p = log 1/p is its "surprise". (If p = 1, log 1/1 = 0, so zero surprise; as p -> 0, the surprise gets bigger and bigger; and the surprise for two independent events, p = p1 * p2, is the sum of their individual surprises: log 1/(p1*p2) = log 1/p1 + log 1/p2.)

The entropy of a distribution is its average surprise: Sum/Integral of p log 1/p.

KL(p || q) is your excess surprise if you think something's distribution is q but it's actually p: Sum/Integral p (log 1/q - log 1/p). The KL divergence is always non-negative because surely if you think the distribution is q but it's actually p, on average you're going to be more surprised than someone who knows it's p.

Re:A Friendly Introduction to Cross-Entropy Loss (2016) by heavenlyblue on Thursday June 07, 2018 @10:46AM (#17256191)
If you're introducing a new term solely for the sake of explaining something, then your fundamentals are wrong.

Bits are fundamental to understanding why we can encode simple numbers in a GNN. If you don't understand that, then surprise-surprise - you need to create another, possibly misleading further down the line, framework.

Re:A Friendly Introduction to Cross-Entropy Loss (2016) by rerx on Thursday June 07, 2018 @03:10AM (#17253886)
The cross entropy can be understood as the expected number of bits you need to specify some event if you assume an underlying distribution. It is large if your assumed distribution is very different from the true distribution. This is indeed helpful intuition for making sense of formulas in machine learning.
Re:A Friendly Introduction to Cross-Entropy Loss (2016) by stared on Thursday June 07, 2018 @04:39AM (#17254278)
Re:A Friendly Introduction to Cross-Entropy Loss (2016) by dangom on Thursday June 07, 2018 @10:47AM (#17256196)
Readers may be interested in checking out this blog post: http://colah.github.io/posts/2015-09-Visual-Information/