I do have a question, but first I must say that as I was reading your post I found myself nodding in agreement again and again and again. Yes, the Silicon-Valley-centric AI research community (including UK-based DeepMind and its surrounding satellites) tends to overlook or even ignore important research from elsewhere. Yes, "algorithmic creativity" and focused tinkering often prove more important than computational resources. Yes, innovation in deep learning does not require big data nor massive hardware, but "engineers are drawn to using the biggest datasets they can get, on the biggest machines they can access, like moths flitting around a bright light" (so true!). And yes, "genuine advances [have] consistently come from doings things differently, not doing things bigger." Ah-men!

With that out of the way, here's my question:

Have you guys tried or managed to achieve learning super-convergence with attention or residual attention models?