Transformers Can Do Arithmetic with the Right Embeddings

Re:Transformers Can Do Arithmetic with the Right Embeddings by wrsh07 on Tuesday May 28, 2024 @10:15AM (#40500955)

This is cool, but special casing digits is unsatisfying.

It makes me think that the authors have correctly identified an issue (positional embeddings) but don't propose a general solution.

I'm not sure if such a thing is possible, but if it is, it would feel more complete. (Fwiw, positional embeddings have had issues for a long time! So a general solution to this would benefit more than just arithmetic. Helpfully, we now have a really good specific example to serve as a baseline for any generalization we seek)

[ Parent | on HN ]

Re:Transformers Can Do Arithmetic with the Right Embeddings by neves on Tuesday May 28, 2024 @10:24AM (#40501029)

but it makes sense to have a different encoding. Mathematics is a completely different language. Maybe we should have more than one class of encodings.

[ Parent | on HN ]

Re:Transformers Can Do Arithmetic with the Right Embeddings by PartiallyTyped on Tuesday May 28, 2024 @03:04PM (#40504295)

There were some recent posts (either here or reddit) supporting the claim that different regions activate when reading programs vs when reading text. If we take that to be true; and squint just enough, one could claim that arithmetic and mathematics should be treated differently to language.

[ Parent | on HN ]

Re:Transformers Can Do Arithmetic with the Right Embeddings by ckemere on Tuesday May 28, 2024 @04:37PM (#40505357)

Numeracy is definitely associated with different brain regions than just reading. See, e.g. https://www.sciencedirect.com/science/article/pii/S105381191...

(Dehaene also has a book, “The Numbet Sense”)

[ Parent | on HN ]

Re:Transformers Can Do Arithmetic with the Right Embeddings by wrsh07 on Tuesday May 28, 2024 @07:34PM (#40506709)

I would only find that satisfying (from a snobbish and impractical perspective) if we were able to have the model decide: 1) what encoding should this section use? 2) how should I train this encoding?

A mixture of experts but for encodings is interesting, though!

Maybe there's a clean way to implement

[ Parent | on HN ]

Re:Transformers Can Do Arithmetic with the Right Embeddings by PeterisP on Wednesday May 29, 2024 @05:00AM (#40509904)

For arbitrary documents and queries, how do we reliably segment the text between those two different languages? And if we can do that, why can't the model do it implicitly?

[ Parent | on HN ]