Re:Improving the way neural networks learn
by michael_nielsen on Friday August 01, 2014 @03:20PM
(#8122277)
This is addressed in the marginal note attached to the sentence you quoted.
The essential point is that we're considering classification problems, for which the output is intended to be 0 or 1. I address the more general case of regression problems (where y may take any value) in a later exercise.
Hope that helps!
The statement "if the neuron's actual output is close to the desired output, i.e., y=y(x) for all training inputs x, then the cross-entropy will be close to zero"
is not true. The function peaks in the middle (~ 0.7)
Thanks! -Kaushik