Re:Deep neural network written from scratch in Julia
by arvinsim on Monday December 07, 2015 @01:05AM
(#10688153)
All of these machine learning news is making me regret not taking math seriously in my undergraduate studies. I am beginning to get bored doing web development.
Re:Deep neural network written from scratch in Julia
by arvinsim on Monday December 07, 2015 @07:16AM
(#10688932)
Thanks! I am already excited on the prospect of relearning math and opening new avenues of knowledge. I am already planning out my "curriculum" for the next year.
Re:Deep neural network written from scratch in Julia
by IshKebab on Monday December 07, 2015 @04:12AM
(#10688493)
The Readme actually clarified a lot of things for me that many longer texts skip over - why ReLU's are used (they prevent vanishing (or exploding) gradients), exactly how dropout works, why there is "momentum", etc.
I'm still not sure why he uses softplus instead of ReLU though. The implication is that it is better to have a smooth function, but is it? And does the benefit outweight the extra computational burden?
Re:Deep neural network written from scratch in Julia
by jostmey on Monday December 07, 2015 @07:23AM
(#10688965)
One could separate the training data into a validation and training set. Then you could try both the ReLU and softplus to see which performs better. I have no idea which type would---I just liked the idea of using a smooth activation function instead of a jagged one.
Re:Deep neural network written from scratch in Julia
by aidanf on Monday December 07, 2015 @09:03AM
(#10689442)
From the README, under the performance section:
> This package is not written for speed. It is meant to serve as a working example of an artificial neural network. As such, there is no GPU acceleration. Training using only the CPU can take days or even weeks. The training time can be shortened by reducing the number of updates, but this could lead to poorer performance on the test data. Consider using an exising machine learning package when searching for a deployable solution.
It seems the main aim of this software is educational, not production use.
Re:Deep neural network written from scratch in Julia
by jostmey on Monday December 07, 2015 @02:55PM
(#10692188)
Actually, I added that disclaimer on performance because of your first comment. I realized people were getting the wrong idea about my little example, and were thinking this could be used in place of packages like Caffe, Torch7, Theano, TensorFlow, ect...
Re:Deep neural network written from scratch in Julia
by kapv89 on Monday December 07, 2015 @12:32PM
(#10690961)
I really hope someone writes a few good machine learning tutorials in Javascript (preferably ES2015). Many people use it for everything else other than machine learning stuff. Wouldn't hurt to not having to shift to another lang when dealing with ML.
Re:Deep neural network written from scratch in Julia
by aman2304 on Monday December 07, 2015 @12:19AM
(#10688075)
With so many deep/machine learning frameworks around, which framework would you put your money on? I use Theano but it makes life difficult when it comes to debugging
Re:Deep neural network written from scratch in Julia
by Veratyr on Monday December 07, 2015 @03:11PM
(#10692311)
Yup but those are the only libraries for Python that'll use the GPU. The rest are mostly CPU-only (PyBrain, quite slow as a result), built on Theano (Lasagne, pydnn), very specific to a particular problem (Caffe) or written in a non-compatible language (Torch).
Given that it supports 2/3 of the big general purpose libraries, it's good enough.
Re:Deep neural network written from scratch in Julia
by olympus on Monday December 07, 2015 @12:00AM
(#10688051)
Short answer: No.
Long answer: Looking at the code, this is written in pure Julia and nothing in place for running on a GPU. You could (re)write it but I'm guessing that's not what you meant when you asked.
Re:Deep neural network written from scratch in Julia
by etrain on Monday December 07, 2015 @12:40AM
(#10688107)
Short answer is actually: maybe!
The bulk of the work done in this code (in terms of FLOPS and, likely, wall-clock time) is going to be in BLAS-3 operations in the feed-forward and back-prop steps. That is, almost all of the work is done using Matrix-Matrix multiplies and in-place arithmetic/transcendental functions.
CUBLAS[1] will allow you to run these types of operations on your GPU at highly accelerated rates, without much more effort than replacing your BLAS library with a new binary. Additionally, if you want finer granularity control over what gets done on the GPU, there are other libraries[2] which provides a direct interface to CUBLAS.
Time to peruse Khan Academy and befriend Math.
I'm still not sure why he uses softplus instead of ReLU though. The implication is that it is better to have a smooth function, but is it? And does the benefit outweight the extra computational burden?
Also, the code is fantastically short.
> This package is not written for speed. It is meant to serve as a working example of an artificial neural network. As such, there is no GPU acceleration. Training using only the CPU can take days or even weeks. The training time can be shortened by reducing the number of updates, but this could lead to poorer performance on the test data. Consider using an exising machine learning package when searching for a deployable solution.
It seems the main aim of this software is educational, not production use.
Given that it supports 2/3 of the big general purpose libraries, it's good enough.
Long answer: Looking at the code, this is written in pure Julia and nothing in place for running on a GPU. You could (re)write it but I'm guessing that's not what you meant when you asked.
Look at Mocha.jl if you want a Neural Network implementation in Julia that can run on a GPU: http://devblogs.nvidia.com/parallelforall/mocha-jl-deep-lear...
The bulk of the work done in this code (in terms of FLOPS and, likely, wall-clock time) is going to be in BLAS-3 operations in the feed-forward and back-prop steps. That is, almost all of the work is done using Matrix-Matrix multiplies and in-place arithmetic/transcendental functions.
CUBLAS[1] will allow you to run these types of operations on your GPU at highly accelerated rates, without much more effort than replacing your BLAS library with a new binary. Additionally, if you want finer granularity control over what gets done on the GPU, there are other libraries[2] which provides a direct interface to CUBLAS.
[1] https://developer.nvidia.com/cublas [2] https://github.com/JuliaGPU/CUBLAS.jl