Re:Why Are Eight Bits Enough for Deep Neural Networks?
by p1esk on Sunday September 20, 2015 @01:49PM
(#10248338)
Interesting, but then how do you explain that rectified linear operations between layers work better than sigmoids?
According to your logic, ReLU should have worse error propagating quality than squashing functions?
The reason NNs don't exhibit strong error propagation is because of the non-linearities between linear layers that perform operations analogous to threshold/majority voting or the like, which have error correction properties.