Re:Hacker's guide to Neural Networks (2012)
by albertzeyer on Friday July 14, 2017 @01:22PM
(#14771460)
The problem is, this really is different on each task. It's really hard to state any generic rules of thumb. Everyone has their own default parameters and intuitions but I would say most of them are heavily biased to the tasks you have worked with. For example, I work with deep bidirectional LSTMs on acoustic modeling for speech recognition, and I use Adam with a starting learning rate of 0.0005, Newbob learning rate scheduling, pretraining, dropout of 0.1 in addition to L2 regularization, and so on. I tried to collect these results here: https://www-i6.informatik.rwth-aachen.de/publications/downlo...
[ Parent
| on HN ]
I know nothing about any of this, but has there been any work into using neural networks to guide and vary the parameter values during training?
Re:Hacker's guide to Neural Networks (2012)
by chriskanan on Friday July 14, 2017 @02:35PM
(#14772160)
There has been some work related to what you are describing:
Learning to learn by gradient descent by gradient descent https://arxiv.org/abs/1606.04474
Related (but less so), there are also some papers about learning neural network architectures:
Designing Neural Network Architectures using Reinforcement Learning https://arxiv.org/abs/1611.02167
Neural Architecture Search with Reinforcement Learning https://arxiv.org/abs/1611.01578
Re:Hacker's guide to Neural Networks (2012)
by frenchie4111 on Friday July 14, 2017 @02:21PM
(#14772045)
This is great, thank you, added it to tonight's reading list.