Re:Transformers struggle with generalizing tasks beyond pre-...
by valine on Tuesday November 07, 2023 @12:24PM
(#38179783)
Going from C++ to python is a little unfair. How many examples would a human need to learn python coming from C++? Its probably more than you can fit a LLM context window.
Re:Transformers struggle with generalizing tasks beyond pre-...
by telotortium on Tuesday November 07, 2023 @12:35PM
(#38179959)
Not with GPT-4-Turbo 128k, at least if attention works as well at that context size as on smaller ones.
Re:Transformers struggle with generalizing tasks beyond pre-...
by rdedev on Tuesday November 07, 2023 @12:49PM
(#38180187)
I know it's unfair compared to a human but I'm more interested in how much it can do. Like what level of leetcode problems can it solve and how well does it use concepts presented in the few shot applications. The whole point is to establish an upper limit on it's generalization power instead of comparing it to a human
Re:Transformers struggle with generalizing tasks beyond pre-...
by sandkoan on Tuesday November 07, 2023 @02:59PM
(#38181983)
For anyone else interested, the paper he's referring to is "The Reversal Curse": https://arxiv.org/abs/2309.12288.
Another important thing to keep in mind is one paper(wish I could remember which one it was) that showed even larger scale llms have trouble understanding that A=B is same as B=A if they have not seen A or B before