https://ai.googleblog.com/2020/01/reformer-efficient-transfo...
TPUv3 is estimated to be 12 or 16nm process node, so the performance of TPU's next versions could still double over the next years (if needed).
At that pace of model research and hardware improvement, I would say we are still in the AI spring.
Yeah, but they also tend to be pretty profitable.
That suggests that the applications are the low-hanging fruit, with far more interesting results still to be discovered.
Being able to perform planning over ‘simple’ domains like Atari games and Go (and not even in the same trained model!) might not seem very comparable to the real thing, but evolutionary history spent the bulk of its time building up the basics—most animals fail most cognitive tasks—so I don't think this is indicative of the progress being misguided, especially given networks-on-GPUs is literally a 10 year old field.
I think MuZero is a clear example of building by principles over random flailing. I get why there does also seem to be the latter, but it's certainly not the whole of it, and anyhow it worked for evolution ;P.
It's really not. The second half of last year alone had MuZero and Megatron-LM, to name just a couple that most scream to me that we are actually progressing towards AGI.
You say ‘When the state-of-the-art is 97% on a task’, but solved tasks are the least interesting tasks.