Re:A Neural Network in 11 Lines of Python (2015)
by amelius on Wednesday October 18, 2017 @02:20PM
(#15501549)
I'm wondering, given a random truth-table with N binary variables and 1 binary output, what is (worst case) the smallest network that can learn it? (In terms of number of parameters).
Re:A Neural Network in 11 Lines of Python (2015)
by fnbr on Wednesday October 18, 2017 @05:31PM
(#15503351)
This is technically true, but I wonder how close you could get to 100% with minimal size. I would expect that you could get somewhere around >98% with a network that's a few megabytes in size.
Re:A Neural Network in 11 Lines of Python (2015)
by dahart on Wednesday October 18, 2017 @04:23PM
(#15502629)
Having gone through this tutorial (which is great!) and several others, I'm curious what is a good second step for the casual neural network learner?
It sounds like there is a growing bag of tricks neural network researchers are discovering to make training practical and stable for large data sets.
One example would be using relu activation - whenever I play with it in a simple tutorial like this one, training seems to explode and fail much more frequently, so I'm guessing either I'm missing another step people use, or there are some extra constraints on initial conditions?
Using a Gaussian for activation in my tutorials has tended to be more stable and converge much faster, but I assume there is a huge downside lurking somewhere to having a non-monotonically increasing function?
What are the tricks of the trade that a weekend warrior should investigate?
Re:A Neural Network in 11 Lines of Python (2015)
by jackthetab on Wednesday October 18, 2017 @07:37PM
(#15504250)
You may want to check out Andrew Ng's Deep Learning Specialization over on Coursera. [1] One of the courses is specifically about hyperparameter tuning and another about structuring your project. There is a lot of practical information scattered across all the courses.
Yes, I'm taking the specialization and having a blast with it. :-)
Re:A Neural Network in 11 Lines of Python (2015)
by T_D_K on Wednesday October 18, 2017 @04:53PM
(#15502947)
Based on the limited amount of information, I'm assuming that by "training explodes" you mean that your gradient descent never reaches a local minimum. Try lowering your learning rate? You may be "stepping over" the minimum.
Re:A Neural Network in 11 Lines of Python (2015)
by SeanDav on Wednesday October 18, 2017 @02:13PM
(#15501484)
You are both right. It is worthwhile knowing that something has been posted before, in order to review old comments if interested and good content is always worth repeating.
In any case, reposting is allowed, for several good reasons, which have been discussed in the past.
What I'm talking about is the size that is required so that the neural net can learn it. This may be different.
https://www.manning.com/books/grokking-deep-learning
It sounds like there is a growing bag of tricks neural network researchers are discovering to make training practical and stable for large data sets.
One example would be using relu activation - whenever I play with it in a simple tutorial like this one, training seems to explode and fail much more frequently, so I'm guessing either I'm missing another step people use, or there are some extra constraints on initial conditions?
Using a Gaussian for activation in my tutorials has tended to be more stable and converge much faster, but I assume there is a huge downside lurking somewhere to having a non-monotonically increasing function?
What are the tricks of the trade that a weekend warrior should investigate?
The assignments are excellent and will let you implement a deephish network from practically scratch, including backprop, optimizers, tuning etc.
[1]: http://cs231n.stanford.edu/index.html
Yes, I'm taking the specialization and having a blast with it. :-)
[1] https://www.coursera.org/learn/neural-networks-deep-learning
https://news.ycombinator.com/item?id=9886555
https://news.ycombinator.com/item?id=11378022
In any case, reposting is allowed, for several good reasons, which have been discussed in the past.