News for hackers. Stuff that matters.
Re:Deep neural network written from scratch in Julia by IshKebab on Monday December 07, 2015 @04:12AM (#10688493)
The Readme actually clarified a lot of things for me that many longer texts skip over - why ReLU's are used (they prevent vanishing (or exploding) gradients), exactly how dropout works, why there is "momentum", etc.

I'm still not sure why he uses softplus instead of ReLU though. The implication is that it is better to have a smooth function, but is it? And does the benefit outweight the extra computational burden?

Also, the code is fantastically short.

Re:Deep neural network written from scratch in Julia by jostmey on Monday December 07, 2015 @07:23AM (#10688965)
One could separate the training data into a validation and training set. Then you could try both the ReLU and softplus to see which performs better. I have no idea which type would---I just liked the idea of using a smooth activation function instead of a jagged one.