Re:Physicists want to rebuild quantum theory from scratch
by f00_ on Monday September 04, 2017 @04:23AM
(#15166605)
Could you explain KL divergence and/or cross entropy to me?
from wikipedia: "In mathematical statistics, the Kullback–Leibler divergence is a measure of how one probability distribution diverges from a second expected probability distribution"
Kullback-Leibler divergence is used a lot in the reinforcement learning setting
"In information theory, the cross entropy between two probability distributions p and q over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set"
Cross Entropy is used as a cost function in neural networks rather than least squares http://neuralnetworksanddeeplearning.com/chap3.html#the_cros...
Re:Physicists want to rebuild quantum theory from scratch
by f00_ on Monday September 04, 2017 @02:27PM
(#15169963)
Also, have you read Norbert Weiner's Cybernetics? I think he had parts of information theory there before Shannon
I want to understand his work on random processes too, have just kind of scanned the book and have tried to upgrade my calculus
"The form of H will be recognized as that of entropy as defined in certain formulations of statistical mechanics where p_i is the probability of a system being in cell i of its phase space. H is then, for example, the H in Boltzmann’s famous H theorem."
where the general form was H = -K\sum p_i * log(p_i)
IIRC, this form is the only kind with the following two properties:
- chain rule -- H(X,Y) = H(X) + H(Y|X)
- maximized for the uniform distribution.