Re:LaMDA is not sentient
by tsimionescu on Monday June 13, 2022 @05:11AM
(#31722954)
> Transformer models are universal function approximators.
You have to be very careful with the wording here. The class of Transformer models is a universal (continuous) function approximator. A single Transformer model can only approximate 1 particular function.
Also, backpropagation/gradient descent are not universal function approximator constructors - they can't be used to construct an ANN that approximates an arbitrary (continuous) function even with arbitrarily many random input/output pairs from that function's domain.
For example, given a function that is f(x) = floor(x) if floor(x) is even; x if floor(x) is odd, defined for any `double x`, there is no guarantee that doing back-propagation over a random sample of (x, f(x)) pairs will yield the correct function approximator for some arbitrary precision (especially if you don't start with the right architecture).
You have to be very careful with the wording here. The class of Transformer models is a universal (continuous) function approximator. A single Transformer model can only approximate 1 particular function.
Also, backpropagation/gradient descent are not universal function approximator constructors - they can't be used to construct an ANN that approximates an arbitrary (continuous) function even with arbitrarily many random input/output pairs from that function's domain.
For example, given a function that is f(x) = floor(x) if floor(x) is even; x if floor(x) is odd, defined for any `double x`, there is no guarantee that doing back-propagation over a random sample of (x, f(x)) pairs will yield the correct function approximator for some arbitrary precision (especially if you don't start with the right architecture).