News for hackers. Stuff that matters.
Re:Transformers Can Do Arithmetic with the Right Embeddings by msoad on Tuesday May 28, 2024 @04:00AM (#40498482)
> how do you argue that these models are not able to reason?

I don't make this argument. Benchmarks like CLUTRR[1] show how poorly LLMs do in reasoning.

[1] https://github.com/facebookresearch/clutrr

Re:Transformers Can Do Arithmetic with the Right Embeddings by Last5Digits on Tuesday May 28, 2024 @06:56AM (#40499407)
There is a difference between poor reasoning and no reasoning. SOTA LLMs correctly answer a significant number of these questions correctly. The likelihood of doing so without reasoning is astronomically small.

Reasoning in general is not a binary or global property. You aren't surprised when high-schoolers don't, after having learned how to draw 2D shapes, immediately go on to draw 200D hypercubes.

Re:Transformers Can Do Arithmetic with the Right Embeddings by wrsh07 on Tuesday May 28, 2024 @10:21AM (#40500993)
Granting that, the original point was that they're not excited about this particular paper unless (for example) it improves the networks' general reasoning abilities.

The problem was never "my llm can't do addition" - it can write python code!

The problem is "my llm can't solve hard problems that require reasoning"