Re:Auto-Regressive Next-Token Predictors Are Universal Learners
by version_five on Friday September 22, 2023 @08:49PM
(#37619634)
I think that's obvious isn't it? Neural networks are universal function approximates, the question is how to make the efficient, either in parameters or computation or whatever, as well as all the usual stuff like encouraging convergence, avoiding big gradients, etc. That's why transformers are popular, nobody thinks they especially can compute a function that other models can't.
[ Parent
| on HN ]
Re:Auto-Regressive Next-Token Predictors Are Universal Learners
by nonrandomstring on Saturday September 23, 2023 @04:37AM
(#37621613)
Yeah I immediately thought, isn't this congruent to a statement about
Turing machines. Sure there are classes of many things that are
computationally equivalent, including computers made of paper-tape,
tin cans and string. Just most of them are horrendously inefficient
and useful only as thought experiments.
I saw this again in audio synthesis but with more nuance. Most methods are equivalent in some crazy limit, but all have a "special" area of most useful effectiveness. For example in theory you can predict a signal of many minutes or hours just using linear prediction (LPC), but only at the cost of a gargantuan parameter space that's less efficient than just sampling the signal.
Nonetheless it is nice to see that researchers are connecting up these dots, even if the pure maths behind it isn't saying anything obviously useful right away. Who knows what insights this might lead to for discovering other new methods of computation.