Re:Deep Learning and Its Implications for Computer Architect...
by sanxiyn on Thursday November 21, 2019 @08:48AM
(#21593910)
Sparsely activated multitask model is a kind of Jeff Dean's hobby horse. It was published in 2017: https://arxiv.org/abs/1701.06538. My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.
Re:Deep Learning and Its Implications for Computer Architect...
by acollins1331 on Thursday November 21, 2019 @09:57AM
(#21594631)
Capsule networks are not failed experiments! Where is this coming from? They merely haven't been applied as much as CNNs or FCNs but there are lots of papers out there where capsule networks outperform those architectures.
Source: my thesis using capsule networks for semantic segmentation of aerial imagery
Re:Deep Learning and Its Implications for Computer Architect...
by XuMiao on Thursday November 21, 2019 @02:21PM
(#21598293)
I like the Capsule idea too. In some way, capsule network is very similar to sparse attention network. It's just the attention normalization is different. Attention is normalized on the inputs, the capsule is normalized on the output. Potentially capsule can yield much cleaner patterns, while patterns generated by attention networks can be overlapping. It's just that capsule is much harder to solve.
The last section, focussing on having a single large sparsely activated model which can accomplish thousands of different tasks by using a selection of internal 'experts' interests me the most.
I suspect this type of model isn't used much today simply because each company using ML only typically has a few problems to solve. If someone like Google, with far more different problems to solve, can get this type of model to work and demonstrate its effectiveness, I think it would be a big step towards solving artificial general intelligence.
Jeff Dean has a lot of respect and influence inside Google, and his ideas tend to get implemented. I'm looking forward to it!