News for hackers. Stuff that matters.
Re:Transformers struggle with generalizing tasks beyond pre-... by vidarh on Tuesday November 07, 2023 @12:17PM (#38179657)
Why are human children so bad at math until they've repetitively trained on how to apply each basic operation over and over and over even after having learned rules that should be simple to apply?

As it happens, conditioning even people to actually apply rules properly tends to take a lot of repetition. How many individual examples of step-by-step working out basic math problems do you think they have been in their training data?

Prompt them the way you'd prompt a child who is sloppy into working step by step and explaining how it applies the rules it has learned, and it will tend to do better. While tokenization might not help, I don't think there's an inherent problem there beyond feeding them enough training data. Whether that's worthwhile vs. having it resort to tools, is another matter.

There are no comments on this comment yet.