Re:Transformers Can Do Arithmetic with the Right Embeddings
by lupire on Tuesday May 28, 2024 @04:16PM
(#40505095)
Humans have the same limitation and use same solution: showing your work and taking notes. There's no blocker here.
[ Parent
| on HN ]
Re:Transformers Can Do Arithmetic with the Right Embeddings
by jhanschoo on Tuesday May 28, 2024 @09:54PM
(#40507541)
There is a distinction. Humans with the use of an unbounded scratchpad can emulate a general-purpose Turing machine and perform general computation given unbounded time. A LLM is still restricted to its context window which is a comparatively extreme limitation of memory. In comparison, our general-purpose computers have so much memory this isn't something we care about for most practical instances of hard problems that we solve with a classical CS algorithm. You can obviously modify LLMs to perform unbounded computation per token (and furnish it with a scratchpad) but afaict commercial LLMs today don't offer that.