News for hackers. Stuff that matters.
Re:Deep Learning Without Poor Local Minima by rudyl313 on Tuesday May 24, 2016 @06:42PM (#11766146)
But the tricks/advancements I mentioned are not changing the function. They changed the initial weights and how the cost function was explored.
Re:Deep Learning Without Poor Local Minima by aab0 on Tuesday May 24, 2016 @07:21PM (#11766394)
But they do change the function. You use RELU now, instead of pretraining.
Re:Deep Learning Without Poor Local Minima by rudyl313 on Tuesday May 24, 2016 @07:26PM (#11766422)
Using ReLU units is a newer advancement and I agree that changing the activation function does change the cost function. However, before Hinton got all excited about ReLU units, he was still showing huge improvements just by using pretraining and later by using dropout, which shouldn't change the cost function.
Re:Deep Learning Without Poor Local Minima by argonaut on Tuesday May 24, 2016 @09:15PM (#11766966)
Dropout helps with convergence/optimization, sure. The existence of a global minimum says nothing about the time required to reach it. Important to note that dropout isn't as common anymore; it's not a huge win.