News for hackers. Stuff that matters.
Re:Transformers struggle with generalizing tasks beyond pre-... by haltist on Tuesday November 07, 2023 @12:16PM (#38179636)
So why can't GPT do arithmetic? At what scale do you suppose it will manage to add numbers?
Re:Transformers struggle with generalizing tasks beyond pre-... by og_kalu on Tuesday November 07, 2023 @12:34PM (#38179943)
GPT-4 can do arithmetic just fine. In fact, despite being severely hobbled by Tokenization (https://arxiv.org/abs/2110.09485), it can perform arithmetic better than you could without a dedicated tool or pad.
Re:Transformers struggle with generalizing tasks beyond pre-... by Der_Einzige on Tuesday November 07, 2023 @12:42PM (#38180060)
Besides your answer on tokenization being right on the mark (gwern even has articles discussing this exact issue - https://gwern.net/gpt-3#bpes), that linked paper is extremely powerful for theoretical thinking about this problem. I literally just left a comment claiming that almost all of human thought is encoded in LLMs, leading almost all tasks to be considered interpolation.

If this is true, I suppose based on what we know of the curse of dimensionality it makes sense. Very thought-provoking work.

Re:Transformers struggle with generalizing tasks beyond pre-... by og_kalu on Tuesday November 07, 2023 @12:49PM (#38180193)
Damn I linked the wrong paper lol.

Meant to link this. More pertinent to my point on Tokenization https://arxiv.org/abs/2310.02989