Read it. Worth a read, especially for those not closely following the machine learning world.

The last section, focussing on having a single large sparsely activated model which can accomplish thousands of different tasks by using a selection of internal 'experts' interests me the most.

I suspect this type of model isn't used much today simply because each company using ML only typically has a few problems to solve. If someone like Google, with far more different problems to solve, can get this type of model to work and demonstrate its effectiveness, I think it would be a big step towards solving artificial general intelligence.

Jeff Dean has a lot of respect and influence inside Google, and his ideas tend to get implemented. I'm looking forward to it!

Sparsely activated multitask model is a kind of Jeff Dean's hobby horse. It was published in 2017: https://arxiv.org/abs/1701.06538. My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.

Capsule networks are not failed experiments! Where is this coming from? They merely haven't been applied as much as CNNs or FCNs but there are lots of papers out there where capsule networks outperform those architectures.

Source: my thesis using capsule networks for semantic segmentation of aerial imagery

I like the Capsule idea too. In some way, capsule network is very similar to sparse attention network. It's just the attention normalization is different. Attention is normalized on the inputs, the capsule is normalized on the output. Potentially capsule can yield much cleaner patterns, while patterns generated by attention networks can be overlapping. It's just that capsule is much harder to solve.

Risky to say, considering deep learning by Hinton was also once called a failed experiment ...

Short of TeX/LaTeX, I can put up with a Google docs paper.

I absolutely cannot stand a MS-Word paper. It's crippling.

Are you saying it's impossible to format a paper in Word to please you? Seems like there's a lot of options to get the look you want.

Well folks who prepare documents in Word, esp. formal documents, more often than not, pick the default rendering of Times New Roman (and I know the default in MS-Word is not TNR).

And a standard MS-Word TNR document looks like crap. I'm sorry.

But I have to admit I have an over-reaction to seeing an MS-Word document after spending more than a decade working with TeX/LaTeX exclusively. So part of the blame goes to me, and actually hurts my ability to keep up with the literature when I start avoid MS-Word papers.

Yeah, the default Word style definitely isn't anything like a typeset document. For fun I downloaded the "Latex.dot" template and some computer modern fonts, with those you can make a first approximation of the Latex look in Word but it's still not the same.

I've never been a fan of the Computer Modern font but the page layout and formating in Latex is certainly nice and just having a standard for scientific papers is a plus.