News for hackers. Stuff that matters.
Re:Neural networks in JavaScript by zackmorris on Friday July 11, 2014 @10:13AM (#8020197)
This is fantastic. I wonder if we've passed a tipping point where some of these fundamental machine learning algorithms will be taught the same way we learned the C++ STL back in the day. Really something like a spam filter should look like a shell script, where you pipe things around and use off the shelf tools, rather than spending inordinate amounts of time trying to implement building blocks (imaging having to write grep before you could even start).

Just out of curiosity, does anyone have a link on NN training parallelization? I’m just hitting paywalls. I’m curious if it’s simple or if there are dependencies between neurons that make it hard. As I see it, the next big hurdles after ubiquitization are distributed training (possibly with GPUs) and empirical data on the best tuning parameters for various problem spaces (which will come largely for free when training happens 5 or 10 orders of magnitude faster than today). Eventually we’ll have an answer as to whether AGI can be composed of basic building blocks or if we’re still missing some fundamental insight.

For example, genetic algorithms are fairly easy to parallelize because by their nature they use large and independent populations. About the time I was graduating college in the late 90s, there was great interest in using them to find the initial weights in NNs to avoid local minima, but then I didn’t hear much about it since. Have there been any breakthroughs?

Re:Neural networks in JavaScript by joeblau on Friday July 11, 2014 @11:20AM (#8020578)

  > the same way we learned the C++ STL back in the day
My school forbid you from using STL. You had to build all of your data structures in freshman/sophomore years and then use those same data structures you built in your sophomore/junior/senior projects. It was really crazy because if you never got your lists, trees, graphs, and hashes working you basically wouldn't make it through the CS program. It sucked at the time, but I'm grateful now.
Re:Neural networks in JavaScript by squeaky-clean on Friday July 11, 2014 @12:46PM (#8021117)
Wow, that sounds very bad to me. It's important to learn how data structures work, but I believe it's more important to learn how to find and re-use existing code/libraries in your work. This is something schools seem to be avoiding and even discouraging (mine didn't take it as far as your school, but they never once covered the standard libraries).

I've tutored students in Java classes who would write a bubble sort every time they needed to sort a List, instead of using Collections.sort, simply because they didn't know Collections.sort existed! Or their own ugly code of looping, indexOf and substring to get values from a CSV instead of using string.split(",") or a premade csv parser (which I know that professor would have been fine with, he would have even encouraged it).

Unless point of the project is to learn how to make one from scratch, you shouldn't be wasting your time writing an alphabetical sorting function when you can be solving novel problems.

Re:Neural networks in JavaScript by joeblau on Friday July 11, 2014 @03:00PM (#8021821)

  > Wow, that sounds very bad to me.
It might, but today, I run into tons of developers that don't understand the fundamentals. I worked with a guy who I affectionally named "Stack Man." He basically used a stack for any list implementation. He didn't understand the difference between a list, a queue, and a stack.

You're teaching your students Java and telling them "Just use Collections.sort" and that's fine, but do you understand when Collections.sort is bad? If you don't understand the underlying algorithm of how a merge sort works, you might be telling them to use something that is worse than could actually be worse than a bubble sort in certain cases.

Also Java wasn't taught to CS students at our school, that was only for BIT (Business Information Technology). Teachers wanted you to understand memory and pointers.

Re:Neural networks in JavaScript by squeaky-clean on Sunday July 13, 2014 @07:10PM (#8028844)
>You're teaching your students Java and telling them "Just use Collections.sort" and that's fine, but do you understand when Collections.sort is bad? If you don't understand the underlying algorithm of how a merge sort works, you might be telling them to use something that is worse than could actually be worse than a bubble sort in certain cases.

When your assignment has to be uploaded in 3 hours, and your dataset is 120 items, Collections.Sort is always better than writing your own sort.

I never said I taught them only to use Collections.sort, as that would be just as bad as never teaching them the standard libraries and only teaching data-structures. You need a healthy balance of both fields of knowledge. Otherwise you end up spitting out either programmers who can use library functions without understanding what's actually going on, or programmers who know all about the underlying methods and logic, but can't write code efficiently and quickly enough to actually help a team.

That's not to say you cannot write code efficiently or quickly. You likely did as most good programmers do and learned on your own time as well. But for students who don't do this, they will be lacking huge amounts of practical knowledge.

Re:Neural networks in JavaScript by cs02rm0 on Friday July 11, 2014 @04:33PM (#8022208)
<i> I worked with a guy who I affectionally named "Stack Man." He basically used a stack for any list implementation. He didn't understand the difference between a list, a queue, and a stack.</i>

That's brilliant, although I'd imagine it could begin to annoy. Slightly obtuse, but I'd have had to get a snippet of this to play every time he checked in a new stack...

https://www.youtube.com/watch?v=FE0XcdM22Yo

Re:Neural networks in JavaScript by coldtea on Saturday July 12, 2014 @01:21PM (#8025136)
>Wow, that sounds very bad to me. It's important to learn how data structures work, but I believe it's more important to learn how to find and re-use existing code/libraries in your work.

If you're trained as a salaried code churner to solve some "pragmatic enterprise problem" yes.

If you're trained as a computer scientist, you have to know how to do stuff properly from first elements, and also how to invent and code similar new stuff, custom tailored to new domains yourself with ease.

You don't get progress in computer science with people just learning to use existing libraries and data structures. At best you can get some innovative apps, but not people able to build the substructure to create the innovative apps of tomorrow.

Not to mention the sorry state of most existing data structures libraries and core APIs, from STL to the Java SDK. A lot of this stuff needs to be burned with fire to be brought to 2014 levels.

Re:Neural networks in JavaScript by nightski on Friday July 11, 2014 @02:17PM (#8021585)
If you are looking for a tutorial on searching the web, college is probably not worth the money.
Re:Neural networks in JavaScript by agibsonccc on Friday July 11, 2014 @11:06AM (#8020472)
We gave a talk on this at hadoop summit recently[1]. This is also what I've implemented in deeplearning4j [2].

The approach called iterative reduce is basically training on mini batches in parallel and averaging the outcomes. The trick is to do it in multiple iterations on each mini batch averaging the parameters at runtime. It's a surprisingly simple technique that works really well.

Unfortunately, it's not as async as sandblaster LBFGS by dean and co[3].

Andrew himself has told me that the gradient was unstable with this approach though.

Right now, distributed training of neural nets is still like the wild west. I can see a few approaches coming up over time though.

[1] https://www.youtube.com/watch?v=h2Ixpfn-DTg

[2] http://deeplearning4j.org/

[3]: http://research.google.com/archive/large_deep_networks_nips2...

Re:Neural networks in JavaScript by ihodes on Friday July 11, 2014 @10:36AM (#8020311)
You may be looking for stochastic gradient descent (SGD). Training neural nets is very parallelizable using such techniques, and can even be done asynchronously (avoiding having to regularly sync different nodes in the cluster) using systems like parameter server.

Geoff Hinton popularized pre-training nets with unsupervised latent-feature learning autoencoders in the 90s, making training nets much faster. More recently GPUs have been used because they perform matrix/vector multiplication much faster. Computer clusters of hundreds and thousands of nodes are used to run SGD at high speeds with the above techniques, and pretraining with autoencoders isn't quite necessary anymore.

Bayesian optimization is one technique which addresses your concern regarding parameter-choosing. Essentially Bayesian optimization uses Bayesian reasoning to plan experiments which seek to discover the most information about the loss function of the network, thus allowing you to efficiently choose hyperparamters of your model (whatever it is).

Neural nets have definitely been seeing breakthroughs since the 90s–the last few years alone have been amazing. Check out "Google Brain", "ImageNet neural net", "Never-ending learning (NELL)", Geoff Hinton (reinvention of nns), Tom Mitchell (online learning, proto-AI), Yan LeCun (convnets), Nando de Freitas (bayes opt). Lots of good stuff out there!

Re:Neural networks in JavaScript by dnautics on Friday July 11, 2014 @02:49PM (#8021763)
I've been playing around with neural nets and find that swarm optimization is particularly effective; you can train high-dimensional neural net spaces with less testing per iteration. I think this works because there's a lot of arbitrary symmetrical choices embedded in the neural net. For example, hidden layer neurons are essentially arbitrarily chosen.

Also, I can't use backpropagation for what I'm trying to do.

Thanks for the suggestions. I had heard of Yann LeCun, I'm trying to do image recognition in an "different" way than CNNs, I'm going to read up on the other people in the list.

Re:Neural networks in JavaScript by ihodes on Friday July 11, 2014 @05:16PM (#8022458)
I hadn't heard of PSO. Looks pretty interesting. What's your use case that you cannot use stochastic gradient descent?
Re:Neural networks in JavaScript by dnautics on Friday July 11, 2014 @07:17PM (#8023045)
I prefer using PSO versus feed-forward gradient descent because emprically for my system, a particle size of ~100 gives comparable results to standard gradient descent, on a 400-or-so neuron neural net, which is ~4x slower.

I can't use backpropagation because some of the output nodes are used to apply a transformation on the input followed by reapplication of the (now transformed) input through the neural net.

Re:Neural networks in JavaScript by ihodes on Saturday July 12, 2014 @01:38AM (#8023803)
That sounds similar to a recurrent neural net? Backprop definitely works on those with some modifications. I just asked some guys at CMU/Google, they suggested looking more at SGD or some other MCMC methods that may have included the word "expert" or maybe were similar to simulated annealing or Gibb's sampling… which I can't recall… I know nothing about it, and I'm totally unqualified to say anything about them, but maybe (hopefully!) some of those words will lead to something useful! The guys I chatted with don't study PSO, so maybe don't know enough to be helpful, but were a bit down on it in general.
Re:Neural networks in JavaScript by platz on Friday July 11, 2014 @08:07PM (#8023181)
The guys at h2o.ai have been parallelizing machine learning algos for their open source project. I guess their major insight is to ship around different steps of the algo to different nodes rather than ship around data. And they avoid the sort step in hadoop too.
Re:Neural networks in JavaScript by pilooch on Saturday July 12, 2014 @05:28AM (#8024109)