Re:Deep neural network written from scratch in Julia
by jostmey on Monday December 07, 2015 @07:23AM
(#10688965)
One could separate the training data into a validation and training set. Then you could try both the ReLU and softplus to see which performs better. I have no idea which type would---I just liked the idea of using a smooth activation function instead of a jagged one.
I'm still not sure why he uses softplus instead of ReLU though. The implication is that it is better to have a smooth function, but is it? And does the benefit outweight the extra computational burden?
Also, the code is fantastically short.