News for hackers. Stuff that matters.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by qwertox on Wednesday January 01, 2025 @01:21PM (#42567918)
As long as an LLM is capable of inserting "9.99 > 10.01?" into an evaluation tool, we're on a good way.

It feels a bit like "if all you have is a hammer, everything looks like a nail", where we're trying to make LLMs do stuff which it isn't really designed to do.

Why don't we just limit LLMs to be an interface to use other tools (in a much more human way) and train them to be excellent at using tools. It would also make them more energy efficient.

But it's OK if we currently try to make them do as much as possible, not only to check where the limits are, but also to gain experience in developing them and for other reasons. We just shouldn't expect them to be really intelligent.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by riffraff on Wednesday January 01, 2025 @01:28PM (#42567973)
> As long as an LLM is capable of inserting "9.99 > 10.01?" into an evaluation tool, we're on a good way

chatgpt will switch to python for some arithmetic with the result that you get floating point math issues when a 8yo will get the result right. I think "switch to a tool" still requires understanding of which tool to get a reliable result, which in turn means understanding the problem. It's an interesting issue.