Re:Deep Learning Without Poor Local Minima
by rudyl313 on Tuesday May 24, 2016 @07:26PM
(#11766422)
Using ReLU units is a newer advancement and I agree that changing the activation function does change the cost function. However, before Hinton got all excited about ReLU units, he was still showing huge improvements just by using pretraining and later by using dropout, which shouldn't change the cost function.
[ Parent
| on HN ]
Dropout helps with convergence/optimization, sure. The existence of a global minimum says nothing about the time required to reach it. Important to note that dropout isn't as common anymore; it's not a huge win.