News for hackers. Stuff that matters.
Re:Deep neural network written from scratch in Julia by arvinsim on Monday December 07, 2015 @01:05AM (#10688153)
All of these machine learning news is making me regret not taking math seriously in my undergraduate studies. I am beginning to get bored doing web development.

Time to peruse Khan Academy and befriend Math.

Re:Deep neural network written from scratch in Julia by ivan_ah on Monday December 07, 2015 @02:35AM (#10688323)
Here's a crash course on high school math for coders: http://minireference.com/static/tutorials/sympy_tutorial.pdf
Re:Deep neural network written from scratch in Julia by arvinsim on Monday December 07, 2015 @07:14AM (#10688922)
Cool! I did not know such a library exists.
Re:Deep neural network written from scratch in Julia by deepGem on Monday December 07, 2015 @01:44AM (#10688228)
Go for it! Khan academy calculus and algebra courses are awesome. Also, Calculus One on Coursera is really good.
Re:Deep neural network written from scratch in Julia by arvinsim on Monday December 07, 2015 @07:16AM (#10688932)
Thanks! I am already excited on the prospect of relearning math and opening new avenues of knowledge. I am already planning out my "curriculum" for the next year.
Re:Deep neural network written from scratch in Julia by IshKebab on Monday December 07, 2015 @04:12AM (#10688493)
The Readme actually clarified a lot of things for me that many longer texts skip over - why ReLU's are used (they prevent vanishing (or exploding) gradients), exactly how dropout works, why there is "momentum", etc.

I'm still not sure why he uses softplus instead of ReLU though. The implication is that it is better to have a smooth function, but is it? And does the benefit outweight the extra computational burden?

Also, the code is fantastically short.

Re:Deep neural network written from scratch in Julia by jostmey on Monday December 07, 2015 @07:23AM (#10688965)
One could separate the training data into a validation and training set. Then you could try both the ReLU and softplus to see which performs better. I have no idea which type would---I just liked the idea of using a smooth activation function instead of a jagged one.
Re:Deep neural network written from scratch in Julia by Ono-Sendai on Monday December 07, 2015 @07:50AM (#10689090)
Why don't these library authors ever attach performance information? How many training runs per second etc..? How many weights processed per second?
Re:Deep neural network written from scratch in Julia by aidanf on Monday December 07, 2015 @09:03AM (#10689442)
From the README, under the performance section:

> This package is not written for speed. It is meant to serve as a working example of an artificial neural network. As such, there is no GPU acceleration. Training using only the CPU can take days or even weeks. The training time can be shortened by reducing the number of updates, but this could lead to poorer performance on the test data. Consider using an exising machine learning package when searching for a deployable solution.

It seems the main aim of this software is educational, not production use.

Re:Deep neural network written from scratch in Julia by Ono-Sendai on Monday December 07, 2015 @10:03AM (#10689873)
Just because it doesn't run on the GPU doesn't mean it can't be fast. At least they acknowledge it's slow.
Re:Deep neural network written from scratch in Julia by jostmey on Monday December 07, 2015 @02:55PM (#10692188)
Actually, I added that disclaimer on performance because of your first comment. I realized people were getting the wrong idea about my little example, and were thinking this could be used in place of packages like Caffe, Torch7, Theano, TensorFlow, ect...
Re:Deep neural network written from scratch in Julia by kapv89 on Monday December 07, 2015 @12:32PM (#10690961)
I really hope someone writes a few good machine learning tutorials in Javascript (preferably ES2015). Many people use it for everything else other than machine learning stuff. Wouldn't hurt to not having to shift to another lang when dealing with ML.
Re:Deep neural network written from scratch in Julia by aman2304 on Monday December 07, 2015 @12:19AM (#10688075)
With so many deep/machine learning frameworks around, which framework would you put your money on? I use Theano but it makes life difficult when it comes to debugging
Re:Deep neural network written from scratch in Julia by jostmey on Monday December 07, 2015 @07:21AM (#10688955)
Author Here: Not Mine! This is an example of how to code it from scratch to see how the algorithms work. For deployment use a major framework!
Re:Deep neural network written from scratch in Julia by Veratyr on Monday December 07, 2015 @12:37AM (#10688101)
Why choose? Write Keras and switch to whichever takes your fancy at the time.
Re:Deep neural network written from scratch in Julia by aman2304 on Monday December 07, 2015 @12:30PM (#10690950)
Doesn't Keras work with TensorFlow and Theano only?
Re:Deep neural network written from scratch in Julia by Veratyr on Monday December 07, 2015 @03:11PM (#10692311)
Yup but those are the only libraries for Python that'll use the GPU. The rest are mostly CPU-only (PyBrain, quite slow as a result), built on Theano (Lasagne, pydnn), very specific to a particular problem (Caffe) or written in a non-compatible language (Torch).

Given that it supports 2/3 of the big general purpose libraries, it's good enough.

Re:Deep neural network written from scratch in Julia by outlace on Monday December 07, 2015 @02:50AM (#10688341)
TensorFlow, because Google
Re:Deep neural network written from scratch in Julia by NonEUCitizen on Sunday December 06, 2015 @11:40PM (#10688011)
Can this take advantage of CUDA ?
Re:Deep neural network written from scratch in Julia by olympus on Monday December 07, 2015 @12:00AM (#10688051)
Short answer: No.

Long answer: Looking at the code, this is written in pure Julia and nothing in place for running on a GPU. You could (re)write it but I'm guessing that's not what you meant when you asked.

Look at Mocha.jl if you want a Neural Network implementation in Julia that can run on a GPU: http://devblogs.nvidia.com/parallelforall/mocha-jl-deep-lear...

Re:Deep neural network written from scratch in Julia by etrain on Monday December 07, 2015 @12:40AM (#10688107)
Short answer is actually: maybe!

The bulk of the work done in this code (in terms of FLOPS and, likely, wall-clock time) is going to be in BLAS-3 operations in the feed-forward and back-prop steps. That is, almost all of the work is done using Matrix-Matrix multiplies and in-place arithmetic/transcendental functions.

CUBLAS[1] will allow you to run these types of operations on your GPU at highly accelerated rates, without much more effort than replacing your BLAS library with a new binary. Additionally, if you want finer granularity control over what gets done on the GPU, there are other libraries[2] which provides a direct interface to CUBLAS.

[1] https://developer.nvidia.com/cublas [2] https://github.com/JuliaGPU/CUBLAS.jl