Re:Kullback–Leibler Divergence
by imjonse on Tuesday August 22, 2023 @10:18AM
(#37223082)
If you look deep enough they are not exclusive. From the deeplearning book:
"Many authors use the term “cross-entropy” to identify specifically the negative log-likelihood of a Bernoulli or softmax distribution, but that is a misnomer. Any loss consisting of a negative log-likelihood is a cross-
entropy between the empirical distribution defined by the training set and the
probability distribution defined by model. For example, mean squared error is the
cross-entropy between the empirical distribution and a Gaussian model"
Re:Kullback–Leibler Divergence
by yobbo on Wednesday August 23, 2023 @06:30AM
(#37233935)
Also, the gradients of "softmax loss" and mean-square-error loss are the same. The network learns to optimize the same function, up until the output activation.
"Many authors use the term “cross-entropy” to identify specifically the negative log-likelihood of a Bernoulli or softmax distribution, but that is a misnomer. Any loss consisting of a negative log-likelihood is a cross- entropy between the empirical distribution defined by the training set and the probability distribution defined by model. For example, mean squared error is the cross-entropy between the empirical distribution and a Gaussian model"
https://stats.stackexchange.com/questions/288451/why-is-mean...