Re:Outperforming LLMs with less training data and smaller mo...
by treprinum on Friday September 22, 2023 @11:03AM
(#37612982)
I agree but I am tired of the argument that "all you need is two layers" I hear from (obsolete) academicians all the time so I pointed out the practice is completely different even if theory says so.
And part of the reason why single-hidden-layer networks aren't enough even in continuous memoryless Euclidean cases is, again, because of how loss functions work; you're unlikely to converge on a good approximation with very few hidden layers.