There is a very common structure to this. Despite neural nets having their own baggage, they're worth understanding.
The structure you're wanting is definitely in there. Edit: Yes a bit of self promotion here. Just making a point with patterns I've found as I've built this out.
See:
https://github.com/agibsonccc/java-deeplearning/blob/master/...
https://github.com/agibsonccc/java-deeplearning/blob/master/...
Half the battle is understanding the linear algebra going on here. Beyond that you can pretty much do everything with one set of algorithms and terminology.
For those who go WTF java are you insane? The core idea I'm linking to here is the fact that deep nets are composed of singular neural networks with slight variations having a very common structure for both the singular layer as well as the deep nets themselves.
I would add that Support Vector Machines also work and they are similar and have a much clearer math to them [1]. But SVM and neural nets are ultimately just linear matchers on a nonlinear pattern space, they ultimately involve adhoc choices that experts learn over time.
As I said above, once you learn the maths (linear algebra, statistics, functional-analysis or whatnot), it become less basic understand and more "understanding how", a series of tweaks that experts "with a feel for this stuff" do. But this "feel" level understanding seems exactly what stands in the way of serious, rational progress on the topic.
If you're looking for a basic example in .NET, I think the XOR one is as simple as it gets.
Deep-Learning XOR: https://github.com/primaryobjects/deep-learning/blob/XOR/Dee...
Deep-Learning Digits: https://github.com/primaryobjects/deep-learning/blob/master/...
https://github.com/jpatanooga/Metronome
We have a wiki and some resources put up.
Is that Word2Vec implementation you have roughly equivalent of the Google version[1]?
Any examples of how to use deeplearning4j generally?
Edit: Poke around in the tests, Here's an example of it learning a compressed version of MNIST: https://github.com/agibsonccc/java-deeplearning/blob/master/...
I have a lot more example usage in each of the tests. Test coverage was a higher priority above the documentation, but example usage is there. I'm more than happy to answer emails around the usage of the library as well. I also take feature requests.
I plan on implementing convolutional nets, recursive neural nets and some other ones based around that same structure; That includes the scale out versions with akka for easy multi threading or clustering ( I have built in service discovery with zookeeper among other things in there.
Matlab/Octave is a great way to practice this due to the native data types. If python is your thing numpy's arrays are also pretty easy to digest.
Subtle little tricks like this: https://www.youtube.com/watch?v=evF-3ykjRU0
And understanding the dynamics of scalar operations vs matrix - vector operations.
The machine learning class has some good fundamentals if you need a refresher on how something works.
There will be more complex things like some optimization algorithms have different uses for eigen values: http://see.stanford.edu/materials/lsocoee364b/11-conj_grad_s...
See: http://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors
One other thing might be understanding different ways you can manipulate data. In this case, numerical representation here is an example per row when I toss in one matrix for training. This is applicable to many machine learning problems.
http://www.inference.phy.cam.ac.uk/itila/book.html
http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf
http://www.gaussianprocess.org/gpml/chapters/RW.pdf
Also, an excellent list of ML resources: http://www.reddit.com/r/MachineLearning/comments/1jeawf/mach...
You are mostly stuck with Bayesian models, since that's what we have general-purpose inference algorithms for. And in practice an understanding of Bayesian statistics is a prerequisite for writing useful probabilistic programs.
Having done a small amount of machine learning, I can see how the advice here is "true". And by "true", I mean appropriate for the way that machine learning exists and operates in present-day space. Algorithms are difficult, temperamental and requires expert "tuning".
The sequence seems to be:
- First you learn the formal theory, the math and statistics.
- Then you learn the "squinting", the ad-hoc rules for how to apply which algorithm.
- Then implement the thing
This works better than just starting your editor and piecing code. However, I would claim that this doesn't actually work well in the sense that this is kind of where AI/ML have bogged down. I mean, there are only 5 main approaches, 20 main algorithms and whatever subsidiaries and random stuff. They don't work great and the only progress is incremental (though there is progress and throwing more computer power around at the same time enhances - while masking the low amount of conceptual progress).
What's lacking is any modularity in combining algorithms. The power of ordinary programming is, essentially, using function calls to put together what you want. ML doesn't do that and for all the magic, that makes it weak and fragile - when one magic algorithm doesn't work well, rather than improving it, it really is better, at present, in the interest of getting stuff done, to start with a different magic algorithm. This is true, I'm a realistic in the sense of accepting the present but an idealist in the sense of saying "that kind of sucks, we should be able to fix problems, not surrender and regroup".
Yes, I'm happy to denigrate the good and proper in my question for the best. But I'm an idealist, I suppose it's a matter of taste.