News for hackers. Stuff that matters.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by elliotto on Wednesday January 01, 2025 @02:18PM (#42568457)
Could you share the exact chat you used for when it failed? There is a share chat button on openai.

It's very difficult to be an AI bull when the goalposts are moving so quickly that ai answering core correctly across multiple models is brushed off as 'nondeterministically getting it correct sometimes'

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by stefan_ on Wednesday January 01, 2025 @03:18PM (#42568932)
Why? Did a grocery store self checkout ever fail to calculate sales tax? Do I need to run a study on that?

The people selling this could not make a car drive but now its AGI.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by scrollaway on Wednesday January 01, 2025 @08:05PM (#42570699)
A single-purpose state machine not failing to do the single thing it was created to do does not make for the clever retort you think it makes.

"AGI": emphasis on "G" for "General". The LLMs are not failing to do generalized tasks, and that they are nondeterministic is not a bug. Just don't use them for calculating sales tax. You wouldn't hire a human to calculate sales tax in their head, so why do you make this a requirement in order to call an LLM "AGI"?

I wonder when the goalposts will stop moving from "We have superhuman intelligences which are able to rather reliably converse in many languages, do generalized tasks and automate operations we thought were impossible to automate 3 years ago" (and by the way, this is what we have TODAY), all the way to "It's not AGI unless it's an omnipotent god that knows how to turn water into wine and calculate the applicable sales tax of that operation".

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by Jensson on Thursday January 02, 2025 @04:33AM (#42573020)
> You wouldn't hire a human to calculate sales tax in their head

Everyone did that 60 years ago, humans are very capable at learning and doing that. Humans built jetplanes, skyscrapers, missiles, tanks, carriers without the help of electronic computers.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by sdenton4 on Thursday January 02, 2025 @10:24AM (#42575207)
Yeah... They used side rules and vast lookup tables of function values printed on dead trees. For the highest value work, they painstakingly built analog calculators. They very carefully checked their work, because it was easy to make a mistake when composing operations.

Humans did those things by designing failsafe processes, and practicing the hell out of them. What we would likely consider over fitting in the llm training context.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by stefan_ on Thursday January 02, 2025 @02:44PM (#42577979)
I do not get it, where are all the serious people? Just come out and say what it is and what it is not. Pretend for a moment you have the ethics and work philosophy of an undergraduate science student.