One-shot Learning with Memory-Augmented Neural Networks

Re:One-shot Learning with Memory-Augmented Neural Networks by bra-ket on Friday May 20, 2016 @05:10PM (#11741545)

If you're interested in this you may want to take a look at SDM content-addressable memory, which uses neural networks as address encoders/decoders: https://en.wikipedia.org/wiki/Sparse_distributed_memory

it was developed by Pentti Kanerva at NASA in the 80s

For related memory-augmented models see: https://en.wikipedia.org/wiki/Deep_learning#Networks_with_se...

There was also "Reasoning, Attention, Memory (RAM)" NIPS Workshop 2015 organized by Jason Weston on this topic: http://www.thespermwhale.com/jaseweston/ram/

There is disproportionate amount of work on training learning models while ignoring memory mechanics. For some reason this research is pursued by very few machine learning/AI labs, mostly at Google DeepMind/Brain, FB, and Numenta, and may be Tom Mitchell's 'never-ending learning' project at CMU.

[ Parent | on HN ]

Re:One-shot Learning with Memory-Augmented Neural Networks by ericjang on Friday May 20, 2016 @05:44PM (#11741767)

I think part of the reason why memory / attention mechanisms haven't caught on quite yet is that they add a layer of complexity to reasoning about what DNNs can/cannot do, which is already challenging for researchers. It takes awhile to prove that these are scalable enough for a "killer app".

Also, industry applications of deep learning are at least 1-2 years behind academia, so I don't expect to see any use of differentiable memory mechanisms in production anytime soon.

Anyway, the workshop reading list looks amazing. Thanks for this!

[ Parent | on HN ]