Re:Ten Lessons from Three Generations Shaped Google’s TPUv4i...
by omegalulw on Tuesday January 04, 2022 @12:08PM
(#29797061)
> But this suggests a real breakthrough is needed to keep pace with future requirements.
Not necessarily. Look at papers like the lottery ticket hypothesis - big ML models may be doing better simply because gradient descent just isn't doing a good enough job. Better optimizers would go a long way than just throwing compute at the the problem. Even if you can, it's impractical to use something like GPT-3 all the time.
Not necessarily. Look at papers like the lottery ticket hypothesis - big ML models may be doing better simply because gradient descent just isn't doing a good enough job. Better optimizers would go a long way than just throwing compute at the the problem. Even if you can, it's impractical to use something like GPT-3 all the time.