Re:AI training is outpacing Moore’s Law
by sdenton4 on Wednesday December 08, 2021 @08:48PM
(#29492872)
I'll throw in a seventh...
Neural architecture search is finding more efficient architectural building blocks which translate to better quality models with fewer overall parameters.
There's a lot of work going on right now to improve transformer architectures in particular; I think once the dust settles they will be massively more efficient than the current models. Efficient transformers often also introduce an inductive bias, which ends up improving model quality. (similar to replacing fully connected networks with convolutional networks; more efficient, but also carrying an inductive bias that improves model quality.)
Re:AI training is outpacing Moore’s Law
by visarga on Thursday December 09, 2021 @01:04AM
(#29494348)
We're also realising we don't need full transformers all the time. Token mixing might suffice, there are quite a few papers replacing the core of the transformer with something simpler.
Neural architecture search is finding more efficient architectural building blocks which translate to better quality models with fewer overall parameters.
There's a lot of work going on right now to improve transformer architectures in particular; I think once the dust settles they will be massively more efficient than the current models. Efficient transformers often also introduce an inductive bias, which ends up improving model quality. (similar to replacing fully connected networks with convolutional networks; more efficient, but also carrying an inductive bias that improves model quality.)