Re:Deep Learning Without Poor Local Minima
by aab0 on Tuesday May 24, 2016 @07:21PM
(#11766394)
But they do change the function. You use RELU now, instead of pretraining.
[ Parent
| on HN ]
Using ReLU units is a newer advancement and I agree that changing the activation function does change the cost function. However, before Hinton got all excited about ReLU units, he was still showing huge improvements just by using pretraining and later by using dropout, which shouldn't change the cost function.
Dropout helps with convergence/optimization, sure. The existence of a global minimum says nothing about the time required to reach it. Important to note that dropout isn't as common anymore; it's not a huge win.