When models commonly achieve 97% on a task, it means it's time to define a harder task, as it's long stopped providing any useful signal.
Depending on which subfield interests you, progress may be slowing or accelerating. That's why another "AI winter" is a bogus and alarmist concept. Winter for whom?
There's an old saying "If all you have is a hammer, all your problems start to look like nails." We may see an AI winter come about simply because we run out of things to pound with our hammer.
It's really not. The second half of last year alone had MuZero and Megatron-LM, to name just a couple that most scream to me that we are actually progressing towards AGI.
You say ‘When the state-of-the-art is 97% on a task’, but solved tasks are the least interesting tasks.
https://ai.googleblog.com/2020/01/reformer-efficient-transfo...
TPUv3 is estimated to be 12 or 16nm process node, so the performance of TPU's next versions could still double over the next years (if needed).
At that pace of model research and hardware improvement, I would say we are still in the AI spring.
Yeah, but they also tend to be pretty profitable.
That suggests that the applications are the low-hanging fruit, with far more interesting results still to be discovered.
Being able to perform planning over ‘simple’ domains like Atari games and Go (and not even in the same trained model!) might not seem very comparable to the real thing, but evolutionary history spent the bulk of its time building up the basics—most animals fail most cognitive tasks—so I don't think this is indicative of the progress being misguided, especially given networks-on-GPUs is literally a 10 year old field.
I think MuZero is a clear example of building by principles over random flailing. I get why there does also seem to be the latter, but it's certainly not the whole of it, and anyhow it worked for evolution ;P.
Your Tesla might be 99.99% in recognizing red lights but will continue to drive straight into paper boxes in its immediate path.
odds of crash a => 97% => 3 / 100
odds of crash for b => 99.99% => 1 / 10000
improvement (odds ratio in this case) is then 300x = odds of crash for b / odds of crash for a
However, state-of-the-art models aren't really used in production yet. I think the trend of "use AI/ML to solve X" has only started to pick up in the past 2 years, and it'll continue well into the 2020s. The process of taking research models and putting them into production is not standardized yet, and many models don't even really work in production - if your model takes a second to do an inference step that's fine for research but maybe not for a real product.
I think in the next decade, on the research side, benchmarks will be beaten less often, and instead there will be more focus on trying out radically new things, understanding weaknesses in current techniques, and finding new measurements that assess those weaknesses. On the industry side, there will still be lots of cool and exciting new achievements as already-known techniques are applied to old problems that haven't been addressed by AI yet.
As an aside, this was the first time in my life that I read the phrase "10s" referring to the 2010-2019. Kind of an odd-feeling moment!
[1] https://twitter.com/yoavgo/status/1151977499259219968