News for hackers. Stuff that matters.
Re:Transformers Can Do Arithmetic with the Right Embeddings by golol on Tuesday May 28, 2024 @06:15AM (#40499170)
As I understand, conceptually they just changed 346 + 23 = ? to (1: 3, 2: 4, 3: 6) + (1: 2, 2: 3) = ? So it is not that much of a specific hack. There could be a broader principle here where something is holding transformers back in a general fashion, and we might be able to improve on the architecture!
Re:Transformers Can Do Arithmetic with the Right Embeddings by ckemere on Tuesday May 28, 2024 @04:51PM (#40505481)
Hopefully 3:3, 2:4, 1:6 and 2:2, 1:3?