Re:Resurgence of Neural Networks
by jghrng on Thursday February 28, 2013 @05:11PM
(#5301449)
All that being said, anyone care to weigh in on the pros/cons of RBMs vs something like a contractive autoencoder?
For some problems, it may be nice to have a generative model as offered by RBMS (although Rifai et al. published a sampling method for contractive auto-encoders recently: http://icml.cc/2012/papers/910.pdf). I feel like with RBMs, you can design models which incorporate prior knowledge more "easily" (you may end up with pretty complex models...), e.g. the conditional RBM, the mean-covariance RBM or the spike & slab RBM. Additionally, there's the deep boltzmann machine that consists of multiple layers that are jointly trained in an RBM-like fashion.
Auto-encoders are straightforward to understand and implement. With contractive terms or denoising, the are powerful feature extractors as well.
But as you already noted, if you "just" want to have a good classifier, I think it pretty much boils down to personal preference since you're going to spend some effort on making these techniques work well on your problem anyway.
For some problems, it may be nice to have a generative model as offered by RBMS (although Rifai et al. published a sampling method for contractive auto-encoders recently: http://icml.cc/2012/papers/910.pdf). I feel like with RBMs, you can design models which incorporate prior knowledge more "easily" (you may end up with pretty complex models...), e.g. the conditional RBM, the mean-covariance RBM or the spike & slab RBM. Additionally, there's the deep boltzmann machine that consists of multiple layers that are jointly trained in an RBM-like fashion.
Auto-encoders are straightforward to understand and implement. With contractive terms or denoising, the are powerful feature extractors as well.
But as you already noted, if you "just" want to have a good classifier, I think it pretty much boils down to personal preference since you're going to spend some effort on making these techniques work well on your problem anyway.