Deep Learning and Its Implications for Computer Architecture and Chip Design

Re:Deep Learning and Its Implications for Computer Architect... by sanxiyn on Thursday November 21, 2019 @08:48AM (#21593910)

Sparsely activated multitask model is a kind of Jeff Dean's hobby horse. It was published in 2017: https://arxiv.org/abs/1701.06538. My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.

[ Parent | on HN ]

Re:Deep Learning and Its Implications for Computer Architect... by acollins1331 on Thursday November 21, 2019 @09:57AM (#21594631)

Capsule networks are not failed experiments! Where is this coming from? They merely haven't been applied as much as CNNs or FCNs but there are lots of papers out there where capsule networks outperform those architectures.

Source: my thesis using capsule networks for semantic segmentation of aerial imagery

[ Parent | on HN ]

Re:Deep Learning and Its Implications for Computer Architect... by XuMiao on Thursday November 21, 2019 @02:21PM (#21598293)

I like the Capsule idea too. In some way, capsule network is very similar to sparse attention network. It's just the attention normalization is different. Attention is normalized on the inputs, the capsule is normalized on the output. Potentially capsule can yield much cleaner patterns, while patterns generated by attention networks can be overlapping. It's just that capsule is much harder to solve.

[ Parent | on HN ]

Re:Deep Learning and Its Implications for Computer Architect... by jeffshek on Thursday November 21, 2019 @06:22PM (#21600853)

Risky to say, considering deep learning by Hinton was also once called a failed experiment ...

[ Parent | on HN ]