News for hackers. Stuff that matters.

A comment on Outperforming LLMs with less training data and smaller model sizes. Parent comment

Re:Outperforming LLMs with less training data and smaller mo... by sandpaper26 on Friday September 22, 2023 @07:06AM (#37610468)
It's not a likely solution given how loss functions work, but in theory a single model could learn to perform exactly the function you describe. When you say "just do X" where X is any function (in this case, a piecewise function), a large enough model could do it.
[ Parent | on HN ]
Re:Outperforming LLMs with less training data and smaller mo... by sandpaper26 on Friday September 22, 2023 @08:01AM (#37610853)
After some reflection, it's maybe more accurate to visualize this in reverse: all expert models see the problem and attempt a solution, and then some "manager" model decides which expert model has the best solution and outputs it.
[ Parent | on HN ]
Re:Outperforming LLMs with less training data and smaller mo... by liquidpele on Friday September 22, 2023 @08:41AM (#37611245)
Until the manager model decides to outsource the expert models.
[ Parent | on HN ]
Re:Outperforming LLMs with less training data and smaller mo... by treprinum on Friday September 22, 2023 @10:08AM (#37612267)
In theory you need two layers to model any function. In practice this is wildly different.
[ Parent | on HN ]
Re:Outperforming LLMs with less training data and smaller mo... by sandpaper26 on Friday September 22, 2023 @10:29AM (#37612548)
Any memoryless continuous function between two Euclidean spaces, I think you mean. The experts-and-manager model would need to be able to do more than that (as do most neural networks).

And part of the reason why single-hidden-layer networks aren't enough even in continuous memoryless Euclidean cases is, again, because of how loss functions work; you're unlikely to converge on a good approximation with very few hidden layers.

[ Parent | on HN ]
Re:Outperforming LLMs with less training data and smaller mo... by treprinum on Friday September 22, 2023 @11:03AM (#37612982)
I agree but I am tired of the argument that "all you need is two layers" I hear from (obsolete) academicians all the time so I pointed out the practice is completely different even if theory says so.
[ Parent | on HN ]