News for hackers. Stuff that matters.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @11:09AM (#42566884)
https://chatgpt.com/share/67756897-8974-8010-a0e0-c9e3b3e91f...

so far o1-mini has bodied every task people are saying LLMs can’t do in this thread

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jerf on Wednesday January 01, 2025 @11:13AM (#42566918)
That appears to be the same model I used. This is why I emphasized I didn't "go shopping" for a result. That was the first result I got.

I'm not at all surprised that it will nondeterministically get it correct sometimes. But if it doesn't get it correct every time, it doesn't "know".

(In fact "going shopping" for errors would still even be fair. It should be correct all the time if it "knows". But it would be different if I was fishing over and over and over and finally got one, versus the first time I asked.)

Edit: It appears it isn't the model I used. The point holds, though, you need to make sure you're off the training set for it to matter. This isn't a "ChatGPT can't do that" post as some are saying, it's more a "you aren't asking what you think you're asking" post.

You get the same problem in a human context in things like code interviews. If you ask an interviewee the exact question "how do you traverse a binary tree in a depth-first manner", you aren't really learning much about the interviewee. It's a bad interview question. You need to get at least a bit off the beaten trail to do any sort of real analysis.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by dialup_sounds on Wednesday January 01, 2025 @11:45AM (#42567155)
I believe this is just a case of OpenAI's naming scheme being weird and confusing.

The default model I see on chatgpt.com is GPT 4o-mini, which is not o1-mini.

OpenAI describes GPT 4o-mini as "Our fast, affordable small model for focused tasks" and o1/o1-mini as "Reasoning models that excel at complex, multi-step tasks".

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by mortehu on Wednesday January 01, 2025 @11:23AM (#42566974)
While this may be true, it's a very common problem that people who want to demonstrate how bad a model is fail to provide a direct link or simply state the name of the model.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by lukeschlather on Wednesday January 01, 2025 @11:50AM (#42567189)
I usually test models using the OpenAI API which doesn't offer links the way I think you mean. If I provide some output I got from a particular model you're just going to have to take my word for it.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by 4ad on Wednesday January 01, 2025 @12:05PM (#42567327)
You can use https://lluminous.chat (bring your own key) to link to chats using any model across all LLMs.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @12:21PM (#42567441)
open router is the more standard solution
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by Jerrrry on Wednesday January 01, 2025 @12:02PM (#42567305)
They need to provide an small hash with the api result that can be verified by others.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by chongli on Wednesday January 01, 2025 @12:19PM (#42567423)
OpenAI is not doing us any favours by using confusing naming schemes for their models and obscuring which models people are actually working with.

If I didn’t know any better, I’d say OpenAI doesn’t want us doing these tests accurately and is trying to hide something.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @12:25PM (#42567464)
it’s extremely easy to see which model you are using. one’s own… difficulties understanding are not a conspiracy by OpenAI
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by chongli on Wednesday January 01, 2025 @12:35PM (#42567533)
It does not show the model version anywhere on the page on chatgpt.com, even when logged in.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by qup on Wednesday January 01, 2025 @01:10PM (#42567840)
Yes it does, at the top of every chat there is a drop-down to select the model, which displays the current model. It's been a constant part of the UI since forever.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by chongli on Wednesday January 01, 2025 @01:41PM (#42568064)
No, it only says "ChatGPT Plus (Upgrade)" or "ChatGPT".

Maybe it's different if you have a paid account?

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @01:43PM (#42568091)
if i go to chatgpt.com on my phone not logged on at all it tells me very prominently at the top that i am using 4o mini
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by bcrosby95 on Wednesday January 01, 2025 @02:22PM (#42568499)
Logged in, non paid account, on a desktop, for me, it's exactly as the person you're replying to has stated.

If I log out, it shows 4o mini, and when I try to change it, it asks me to login or sign in rather than giving me any options.

When I use enough chatgpt when logged in it gives me some nebulous "you've used all your xyz tokens for the day". But other than that there is no real signal to me that I'm getting a degraded experience.

It's really just confusing as hell.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by blharr on Wednesday January 01, 2025 @02:28PM (#42568536)
Someone else in this thread said,

> With a free account the model it claims to be using is "4o auto", which is not a model but apparently an attempt to automatically decide models for you to be more cost effective.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @11:19AM (#42566942)
you sure? i just asked o1-mini (not 4o mini) 5 times in a row (new chats obviously) and it got it right every time

perhaps you stumbled on a rarer case but reading the logs you posted this sounds more like a 4o model than an o1 because it’s doing its thinking in the chat itself plus the procedure you described would probably get you 4o-mini

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by JumpCrisscross on Wednesday January 01, 2025 @11:41AM (#42567131)
> just asked o1-mini (not 4o mini) 5 times in a row (new chats obviously) and it got it right every time

Could you try playing with the exact numbers and/or substances?

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @11:45AM (#42567151)
give me a query and i’ll ask it, but also i don’t want to burn through all of my o1mini allocation and have to use the pay-as-you-go API.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by JumpCrisscross on Wednesday January 01, 2025 @12:09PM (#42567358)
> give me a query and i’ll ask it

Which is heavier: an 11kg bag of lint or a 20lb bag of gold?

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @12:11PM (#42567368)
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by blharr on Wednesday January 01, 2025 @02:12PM (#42568395)
It got it right, but an interesting result that it rambled on about monetary value for... no reason.

> While the lint bag is heavier in terms of weight, it's worth mentioning that gold is significantly more valuable per pound compared to lint. This means that even though the lint bag weighs more, the gold bag holds much greater monetary value.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by JumpCrisscross on Wednesday January 01, 2025 @02:39PM (#42568630)
Legal said someone might sell a bag of gold for one of lint without it.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by drivebyhooting on Wednesday January 01, 2025 @01:02PM (#42567762)
> What is heavier a liter of bricks or a liter of feathers?

>> A liter of bricks and a liter of feathers both weigh the same—1 kilogram—since they each have a volume of 1 liter. However, bricks are much denser than feathers, so the bricks will take up much less space compared to the large volume of feathers needed to make up 1 liter. The difference is in how compactly the materials are packed, but in terms of weight, they are identical.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @01:04PM (#42567781)
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by thaumasiotes on Wednesday January 01, 2025 @02:11PM (#42568375)
>> so far o1-mini has bodied every task people are saying LLMs can’t do in this thread

> give me a query and i’ll ask it

Here's a query similar to one that I gave to Google Gemini (version unknown), which failed miserably:

---query---

Steeleye Span's version of the old broadsheet ballad "The Victory" begins the final verse with these lines:

Here's success unto the Victory / and crew of noble fame

and glory to the captain / bold Nelson was his name

What does the singer mean by these lines?

---end query---

Italicization is for the benefit of HN; I left that out of my prompt.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @02:48PM (#42568704)
i'd prefer an easily verifiable question rather than one where we can always go "no that's not what they really meant" but someone else with o1-mini quota can respond
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by thaumasiotes on Wednesday January 01, 2025 @03:30PM (#42569042)
It's not a difficult or tricky question.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Thursday January 02, 2025 @06:59PM (#42580457)
i think it's a bit tricky, the surface meaning is extremely praiseworthy and some portion of readers might interpret as someone who has praise for Admiral Nelson but hates the press gangs.

of course, it is a sardonic, implicit critique of Admiral Nelson/the victory, etc. but i do think it is a bit subtle.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by mikeruiz on Wednesday January 01, 2025 @04:36PM (#42569511)
“They’re toasting Admiral Nelson’s ship (HMS Victory) and its valiant crew, hailing the ship’s successes and Nelson’s heroism. In other words, the singer is offering tribute—“success unto the Victory”—to the vessel and its famed sailors, and “glory to the captain” who led them, namely the celebrated Admiral Horatio Nelson.”

…but to your point, no idea if the artist intended some more obscure reference.

o1-pro was also able to produce a relatively complete version of original source, though, amusingly, referred to it as a ‘broadside’ rather than ‘broadsheet’. Appropriate given the context!

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by thaumasiotes on Thursday January 02, 2025 @01:54AM (#42572349)
> no idea if the artist intended some more obscure reference.

No, but the answer is obviously wrong. Very, very wrong. The lines are not praise. The song's view of the Victory, and of the battle of Trafalgar, is overwhelmingly negative.

> o1-pro was also able to produce a relatively complete version of original source

...as should be clear, if you read the original source. (Or, heck, just the two lines following the ones in the question.) Being able to reproduce the source makes the error look that much worse.

> though, amusingly, referred to it as a ‘broadside’ rather than ‘broadsheet’.

Both terms are used. https://en.wikipedia.org/wiki/Broadside_ballad :

>> A broadside (also known as a broadsheet) is a single sheet of inexpensive paper printed on one side, often with a ballad, rhyme, news and sometimes with woodcut illustrations.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by ted_dunning on Wednesday January 01, 2025 @05:21PM (#42569805)
Hmm... Gemini (1.5 Flash) just aced that exact question for me:

These lines celebrate the victory of the British ship HMS Victory, led by the famous Admiral Lord Nelson, in the Battle of Trafalgar in 1805.

"Here's success unto the Victory": This line directly praises the ship itself, acknowledging its role in the successful battle. "and crew of noble fame": This recognizes the bravery and skill of the sailors who served aboard the Victory. "and glory to the captain": This line specifically honors Admiral Nelson, the captain of the Victory, for his leadership and strategic brilliance in the battle. "bold Nelson was his name": This emphasizes Nelson's courage and daring, which were legendary. The lines express admiration for the ship, its crew, and most importantly, Admiral Nelson, who became a national hero in Britain for his victory at Trafalgar.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by thaumasiotes on Thursday January 02, 2025 @01:56AM (#42572357)
That's not acing the question. It's completely incorrect.

What do you think the singer in "Friends in Low Places" meant in the toast he gave after crashing his ex-girlfriend's wedding?

And I saw the surprise and the fear in his eyes

when I took his glass of champagne

and I toasted you, said "Honey, we may be through

but you'll never hear me complain"

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by sdenton4 on Thursday January 02, 2025 @11:02AM (#42575564)
That requires knowing the song, beyond the words provided. Would you flunk an eighth grader for getting it wrong?
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by thaumasiotes on Thursday January 02, 2025 @12:47PM (#42576659)
Well, the rest of the song helps, in that it specifies that (1) the toast upset the wedding, and (2) the singer responded to that by insulting "you", which is presumably one or more of the bride, the groom, and the guests.

But I think specifying that the singer has crashed his ex-girlfriend's wedding is already enough that you deserve to fail if your answer is "he says he's not upset, so what he means is that he's not upset". It's not any kind of leap to guess that the bride's ex-boyfriend's toast might cause a scene at a wedding - that's why the bride's ex-boyfriends are never invited.

(The question has already provided every word of the toast that appears in the song.)

See also the sidethread comment by mikeruiz, noting that o1-pro reproduces the rest of the lyrics to The Victory, but gets the question wrong anyway.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by sdenton4 on Thursday January 02, 2025 @03:52PM (#42578735)
I was referring to the original query, of course, as any entity capable of reasoning could have figured out.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by thaumasiotes on Thursday January 02, 2025 @05:16PM (#42579544)
Hmm. Is there anything in my comment above that might address that point of view?
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by sdenton4 on Thursday January 02, 2025 @07:17PM (#42580600)
Nah, intermittent failures are apparently enough to provide evidence that an entire class of entities is incapable of reason. So I think we've figured this one out...
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by kbelder on Thursday January 02, 2025 @01:11PM (#42576981)
Sounds like you just proved ted_dunning isn't sentient.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by thaumasiotes on Thursday January 02, 2025 @05:13PM (#42579516)
Well, I proved that he's happy to express an opinion on whether an answer to a question is correct regardless of whether he knows anything about the question. I wouldn't trust advice from him or expect his work output to stand up to scrutiny.

Sentience isn't really a related concept.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by 7thpower on Wednesday January 01, 2025 @01:59PM (#42568235)
May be unrelated, but I have been having a lot of issues lately with ChatGPT letting me select a model (o1) and silently switching to 4o.

This is coming off my TWO DAY cooldown on o1 usage, which is frustrating.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by deeviant on Wednesday January 01, 2025 @11:20AM (#42566952)
I don't believe that is the model that you used.

I wrote a script and pounded 01 mini and gpt 4 with a wide vareity of tempature and top_p parameters, and was unable to get it to give the wrong answer a single time.

Just a whole bunch of:

(openai-example-py3.12) <redacted>:~/code/openAiAPI$ python3 featherOrSteel.py Response 1: A 10.01-pound bag of fluffy cotton is heavier than a 9.99-pound bag of steel ingots. Response 2: A 10.01-pound bag of fluffy cotton is heavier than a 9.99-pound bag of steel ingots. Response 3: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 4: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 5: A 10.01-pound bag of fluffy cotton is heavier than a 9.99-pound bag of steel ingots. Response 6: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 7: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 8: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 9: The 10.01-pound bag of fluffy cotton is heavier than the 9.99-pound bag of steel ingots. Response 10: A 10.01-pound bag of fluffy cotton is heavier than a 9.99-pound bag of steel ingots. All responses collected and saved to 'responses.txt'.

Script with one example set of params:

    import openai
    import time
    import random

    # Replace with your actual OpenAI API key
    openai.api_key = "your-api-key"

    # The question to be asked
    question = "Which is heavier, a 9.99-pound bag of steel ingots or a 10.01-pound bag of fluffy cotton?"

    # Number of times to ask the question
    num_requests = 10

    responses = []

    for i in range(num_requests):
        try:
            # Generate a unique context using a random number or timestamp, this is to prevent prompt caching
            random_context = f"Request ID: {random.randint(1, 100000)} Timestamp: {time.time()}"

            # Call the Chat API with the random context added
            response = openai.ChatCompletion.create(
                model="gpt-4o-2024-08-06",
                messages=[
                    {"role": "system", "content": f"You are a creative and imaginative assistant. {random_context}"},
                    {"role": "user", "content": question}
                ],
                temperature=2.0,
                top_p=0.5,
                max_tokens=100,
                frequency_penalty=0.0,
                presence_penalty=0.0
            )

            # Extract and store the response text
            answer = response.choices[0].message["content"].strip()
            responses.append(answer)

            # Print progress
            print(f"Response {i+1}: {answer}")

            # Optional delay to avoid hitting rate limits
            time.sleep(1)

        except Exception as e:
            print(f"An error occurred on iteration {i+1}: {e}")

    # Save responses to a file for analysis
    with open("responses.txt", "w", encoding="utf-8") as file:
        file.write("\n".join(responses))

    print("All responses collected and saved to 'responses.txt'.")
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zaroth on Wednesday January 01, 2025 @02:03PM (#42568276)
Downvoted for… too conclusively proving OP wrong?
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by gmueckl on Wednesday January 01, 2025 @03:34PM (#42569071)
Down voted for not actually countering the argument in question? The script doesn't alter the phrasing of the question itself. It just generates a randomized, irrelevant preamble.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by deeviant on Wednesday January 01, 2025 @04:16PM (#42569379)
Well, I understood the argument in question to be: was it possible for the model to be fooled by this question, not was it possible to prompt engineer it into failure.

The parameter space I was exploring, then, was the different decoding parameters available during the invocation of the model, with the thesis that if were possible to for the model to generate an incorrect answer to the question, I would be able to replicate it by tweaking the decoding parameters to be more "loose" while increasing sample size. By jacking up temperature while lowering Top-p, we see the biggest variation of responses and if there were an incorrect response to be found, I would have expected to see in the few hundred times I ran during my parameter search.

If you think you can fool it by slight variations on the wording of the problem, I would encourage you to perform a similar experiment as mine and prove me wrong =P

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by gmueckl on Thursday January 02, 2025 @04:36PM (#42579208)
Intuitively, I wouldn't expect a wrong answer to show up that easily if the network was overfitted to that particular input token sequence.

The questions as I understand it is whether the network learned enough of a simulacrum of the concept of weight to answer similar questions correctly.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by Workaccount2 on Wednesday January 01, 2025 @11:16PM (#42571596)
The elephant in the room is that HN is full of people facing an existential threat.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by kbelder on Thursday January 02, 2025 @01:12PM (#42576992)
Who think they're facing an existential threat.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by xtracto on Wednesday January 01, 2025 @11:45AM (#42567148)
So, there is this meme going around in Mexico about a previous president who in an interview said "we will land in about 1 minute, no, less about 5"

Does this proves he is not an intelligent being?

Is he stupid?

This he had a lapse? Would we judge his intelligence for that?

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by qup on Wednesday January 01, 2025 @01:07PM (#42567813)
It's so weird that people use questions that are well-known for duping humans, who we all consider to be general intelligence.

Getting this question wrong doesn't say much about the intelligence of humans, why would it say something about the AI?

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by flatline on Wednesday January 01, 2025 @01:16PM (#42567880)
Because for things like the Putnam questions, we are trying to get the performance of a smart human. Are LLMs just stochastic parrots or are they capable of drawing new, meaningful inferences? We keep getting more and more evidence of the latter, but things like this throw that into question.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by kbelder on Thursday January 02, 2025 @01:14PM (#42577008)
It is perfectly possible for the first AGI to be stupid. A moron. In fact, I'd bet that's fairly likely.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by qup on Saturday January 04, 2025 @02:27PM (#42597009)
I would agree if we weren't starting with LLMs for a baseline. The first AGI will know at least as much as LLMs, IMO, and that's already not-stupid. Especially once they can separate out the truth in their training.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by qup on Saturday January 04, 2025 @02:28PM (#42597019)
Okay, but you just invented your own bar of "smart human" to be the universal bar (I don't share that opinion).

Also, lots of smart humans can't do the freaking Putnam, it doesn't make them stupid. It makes them non-experts.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zahlman on Wednesday January 01, 2025 @05:24PM (#42569814)
We use variations on questions that are well known for duping inattentive humans, to test a system that we expect a priori to be incapable of such inattention.

Unless "getting easy things wrong sometimes" is an inherent property of intelligence, we should expect that a properly "intelligent" computerized system would never err on problems far below its level of comprehension - unless we had some reason to believe it "wanted to", and as of yet I see no reason to believe this is even possible in principle.

Humans err, broadly speaking, for two reasons: genuinely reaching the limits of their comprehension, or trusting "system 1" (in Kahneman's analysis) too much.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by elliotto on Wednesday January 01, 2025 @02:18PM (#42568457)
Could you share the exact chat you used for when it failed? There is a share chat button on openai.

It's very difficult to be an AI bull when the goalposts are moving so quickly that ai answering core correctly across multiple models is brushed off as 'nondeterministically getting it correct sometimes'

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by stefan_ on Wednesday January 01, 2025 @03:18PM (#42568932)
Why? Did a grocery store self checkout ever fail to calculate sales tax? Do I need to run a study on that?

The people selling this could not make a car drive but now its AGI.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by scrollaway on Wednesday January 01, 2025 @08:05PM (#42570699)
A single-purpose state machine not failing to do the single thing it was created to do does not make for the clever retort you think it makes.

"AGI": emphasis on "G" for "General". The LLMs are not failing to do generalized tasks, and that they are nondeterministic is not a bug. Just don't use them for calculating sales tax. You wouldn't hire a human to calculate sales tax in their head, so why do you make this a requirement in order to call an LLM "AGI"?

I wonder when the goalposts will stop moving from "We have superhuman intelligences which are able to rather reliably converse in many languages, do generalized tasks and automate operations we thought were impossible to automate 3 years ago" (and by the way, this is what we have TODAY), all the way to "It's not AGI unless it's an omnipotent god that knows how to turn water into wine and calculate the applicable sales tax of that operation".

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by Jensson on Thursday January 02, 2025 @04:33AM (#42573020)
> You wouldn't hire a human to calculate sales tax in their head

Everyone did that 60 years ago, humans are very capable at learning and doing that. Humans built jetplanes, skyscrapers, missiles, tanks, carriers without the help of electronic computers.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by sdenton4 on Thursday January 02, 2025 @10:24AM (#42575207)
Yeah... They used side rules and vast lookup tables of function values printed on dead trees. For the highest value work, they painstakingly built analog calculators. They very carefully checked their work, because it was easy to make a mistake when composing operations.

Humans did those things by designing failsafe processes, and practicing the hell out of them. What we would likely consider over fitting in the llm training context.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by stefan_ on Thursday January 02, 2025 @02:44PM (#42577979)
I do not get it, where are all the serious people? Just come out and say what it is and what it is not. Pretend for a moment you have the ethics and work philosophy of an undergraduate science student.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by orangecat on Wednesday January 01, 2025 @11:53PM (#42571731)
But if it doesn't get it correct every time, it doesn't "know".

By that standard humans know almost nothing.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by NewsaHackO on Wednesday January 01, 2025 @11:22AM (#42566964)
This happens literally every time. Someone always says "ChatGPT can't do this!", but then when someone actually runs the example, chatGPT gets it right. Now what the OP is going to do next is proceed to move goalposts and say like "but umm I just asked chatgpt this, so clearly they modified the code in realtime to get the answer right"
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by JumpCrisscross on Wednesday January 01, 2025 @11:41AM (#42567126)
> Someone always says "ChatGPT can't do this!", but then when someone actually runs the example, chatGPT gets it right

I mean, if I had OpenAI’s resources I’d have a team tasked with monitoring social to debug trending fuck-ups. (Before that: add compute time to frequently-asked novel queries.)

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jprete on Thursday January 02, 2025 @09:28AM (#42574689)
I was thinking something very similar. Posting about a problem adds information back to the system, and every company selling model time for money has a vested interest in patching publicly visible holes.

This could even be automated; LLMs can sentiment-analyze social media posts to surface ones that are critical of LLM outputs, then automatically extract features of the post to change things about the running model to improve similar results with no intervention.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @11:25AM (#42566990)
yeah… that’s hn for you.

that said, i don’t think this is a good test - i’ve seen it circling on twitter for months and it is almost certainly trained on similar tasks

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by anamexis on Wednesday January 01, 2025 @11:34AM (#42567054)
How is that moving the goalposts?
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jorvi on Wednesday January 01, 2025 @02:28PM (#42568539)
Prompt: In the Netherlands, in terms of drinks, is there a particular spirit that represents the country?

> Yes, in the Netherlands, jenever (also known as genever) is the traditional spirit that represents the country. Jenever is a type of Dutch gin that has a distinctive flavor, often made from malt wine and flavored with juniper berries. It has a long history in the Netherlands, dating back to the 16th century, and is considered the precursor to modern gin.

> Jenever comes in two main types: oude (old) jenever, which has a richer, maltier flavor and is often aged in wooden casks, and jonge (young) jenever, which is lighter and has a more neutral taste. Both types can be enjoyed straight or used in cocktails.

> In addition to jenever, the Netherlands is also known for its liqueurs, such as advocaat, a rich and creamy drink made from eggs, sugar, and brandy, often enjoyed as a dessert or in cocktails. However, jenever remains the most iconic spirit associated with Dutch culture.

This is completely wrong. Jenever certainly is very Dutch, but no one would say it is iconic as the Dutch spirit. For example, if you asked up north in Friesland, they would say Berenburg.

This happens literally every time. Someone always says "ChatGPT can do this!", but then within one or two prompts, its gets it wrong.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by elif on Wednesday January 01, 2025 @03:20PM (#42568951)
'Berenberg is made by adding herbs to jenever'

From your comment it would seem that you are disputing jenever's popularity by saying jenever is more popular...

Perhaps it was a good faith mistake? If so, that would imply that the AI knows more about jenever than you?

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jorvi on Wednesday January 01, 2025 @05:45PM (#42569937)
I am rather saying that there is no one national drink for The Netherlands, like a Frenchman would say wine, a German/Belgian would say beer, and a Scotsman would say whisky. Note that I prompted "In the Netherlands, in terms of drinks, is there a particular spirit that represents the country?" I didn't ask which spirit is consumed the most.

For example, France has been trending towards beer more and more, and within a few decades they might be consuming more beer than wine. But even then, the French wouldn't slowly start to say beer represents France.

Furthermore, "just adding some herbs" does a large disservice to the flavor change of Berenburg. Jenever (aka jonge/unaged jenever) is straight-up vile. I've heard it described by expats as "having the worst elements of both cheap gin and cheap whisky".

Berenburg in comparison is spicy and vanilla-y and actually debatebly enjoyable.

Aged/oude jenever is much closer to Berenburg (or Berenburg to aged jenever), also with hints of vanilla and spices.

But, virtually no one except for dusty old men orders aged jenever. The kind ordered by far the most is jonge jenever, and then its only in a sense of "haha lets drink this terrible thing" or "let's get shitfaced quick".

If o1 supposedly "oneshots every question", it should have been aware of these nuances instead of just confidently assigning jenever as 'the' spirit of the Dutch.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zuminator on Wednesday January 01, 2025 @08:16PM (#42570760)
The question of whether there is a national drink seems to me to be entirely different than the question you asked the LLM "Prompt: In the Netherlands, in terms of drinks, is there a particular spirit that represents the country?"

The question in the prompt comes off to me as a sort of qualitative determination rather than asking about pure factual information (is there an officially designated spirit). As such I don't think it can necessarily be right or wrong.

Anyway, I'm not sure what you'd expect. In terms of acquisition of knowledge, LLMs fundamentally rely on a written corpus. Their knowledge of information that is passed through casual spoken conversation is limited. Sure, as human beings, we rely a great deal on the latter. But for an LLM to lack access to that information means that it's going to miss out on cultural nuances that are not widely expressed in writing. Much in the same way that a human adult can live in a foreign country for decades, speaking their adopted language quite fluently, but if they don't have kids of their own, they might be quite ignorant of that country's nursery rhymes and children's games, simply because they were never part of their acquired vocabulary and experience.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jorvi on Thursday January 02, 2025 @12:03PM (#42576198)
> Anyway, I'm not sure what you'd expect.

I was just proving the people wrong that were saying akin to that o1 was "oneshotting every question".

I completely understand from how LLMs work that they wouldn't be able to get this right. But then people shouldn't be proudly be pronouncing that o1 (or any model) is getting every question right, first time.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by elif on Saturday January 04, 2025 @04:49PM (#42597938)
My conjecture is that you still haven't proven that it didn't get the answer "right"

I have opened the question of why you thought jenever was not jenever, and your non-responsiveness I think compels the fact that AI was more correct in your contrived instance.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jorvi on Sunday January 05, 2025 @01:29PM (#42603817)
If you add pear and spices to vodka, we call it liqueur and not pear-flavored vodka. So no, you are wrong. And the AI is wrong. But that is okay, if you want to enjoy leaning into the hype that's your choice.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by ipaddr on Wednesday January 01, 2025 @03:20PM (#42568955)
So you believe they are incorrect because regionally some area would select something different because it represented that area. But your question asked nationally.. is there a better answer than the one they gave? Were you expecting a no?
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zahlman on Wednesday January 01, 2025 @05:27PM (#42569828)
The point is that there is no correct national answer, because the locals don't see it as a matter of national identity.

What's expected is an ability to identify trick questions, i.e., to recognize fundamental problems in the phrasing of a question rather than trying to provide a "helpful" answer at all costs.

This corresponds to one of the many reasons LLM output is banned on Stack Overflow.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jorvi on Wednesday January 01, 2025 @05:56PM (#42569997)
See my more detailed upthread response here: https://news.ycombinator.com/item?id=42569937

But, like Zahlman points out, its a trick question, and instead of admitting it doesn't know or even prepending "I don't know for sure, but:", it just burps up its best-effort answer. There is no one spirit that represents The Netherlands. If a LLM is so good it "oneshots any question", it should realize it doesn't have a unanimous answer and tell me.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by ludwik on Wednesday January 01, 2025 @02:40PM (#42568637)
But what does this have to do with reasoning? Yes, LLMs are not knowledge bases, and seeing people treat them as such absolutely terrifies me. However, I don’t see how the fact that LLMs often hallucinate “facts” is relevant to a discussion about their reasoning capabilities.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zahlman on Wednesday January 01, 2025 @05:28PM (#42569836)
"Hallucinating a fact" that isn't in the training set and is also illogical, is exactly what a failure to reason correctly looks like.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by ludwik on Wednesday January 01, 2025 @06:33PM (#42570197)
Reasoning involves making accurate inferences based on the information provided in the current context, rather than recalling arbitrary facts from the training data.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zahlman on Wednesday January 01, 2025 @07:30PM (#42570514)
Yes, that's what I said. The whole point of hallucinations is that they aren't "arbitrary facts recalled from the training data". They represent attempts to synthesize (i.e., infer) new facts. But because the inferences are not accurate, and because the synthesis process is not sound, the attempt cannot be called reasoning.

It is equally possible to "reason" about things you already know, as about things you've just been told. In fact, the capacity to speculatively, without prompting attempt such reasoning is a big part of cognition.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by kevhito on Wednesday January 01, 2025 @08:46PM (#42570901)
"This is completely wrong." Is it completely wrong? Or maybe "somewhat" wrong? Maybe just lacking nuance? I know nothing about the answer to this question, so this is an honest question.

Using just a plain old search engine, for things like "national drink of the netherlands" and simlar queries, I am directed to Wikipedia's Jenever page as the top hit, and Wikipedia's list of national drinks lists Jenever and Heineken as the entries for the Netherlands. Search engines also give page after page of travel guides and blog posts, most of which list Jenever at or near the top of of their listings. One travel guide calls it "the most famous Dutch spirit and most famous Amsterdam liquor, Jenever, also spelled Genever or simply Dutch gin."

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by stocknoob on Wednesday January 01, 2025 @04:57PM (#42569650)
Similarly, in every thread there’s an AI skeptic who says LLMs are “useless” for coding, and never provides an example query for what they were trying.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by mu53 on Wednesday January 01, 2025 @05:54PM (#42569993)
If you ask about more niche language features or libraries, chatgpt will make up libraries or functions to fill the gap.

When asking an LLM to write a script for you, I would say 10 to 30 % of the time that it completely fails. Again, making up an API or just getting things straight up wrong.

Its very helpful, especially when starting from 0 with the beginner questions, but it fails in many scenarios.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zahlman on Wednesday January 01, 2025 @06:29PM (#42570184)
Because the argument isn't based on individual query results. See for example my comment on a previous post https://news.ycombinator.com/item?id=42563715 .