Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by ipaddr on Wednesday January 01, 2025 @03:20PM
(#42568955)
So you believe they are incorrect because regionally some area would select something different because it represented that area. But your question asked nationally.. is there a better answer than the one they gave? Were you expecting a no?
[ Parent
| on HN ]
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by zahlman on Wednesday January 01, 2025 @05:27PM
(#42569828)
The point is that there is no correct national answer, because the locals don't see it as a matter of national identity.
What's expected is an ability to identify trick questions, i.e., to recognize fundamental problems in the phrasing of a question rather than trying to provide a "helpful" answer at all costs.
This corresponds to one of the many reasons LLM output is banned on Stack Overflow.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by jorvi on Wednesday January 01, 2025 @05:56PM
(#42569997)
See my more detailed upthread response here: https://news.ycombinator.com/item?id=42569937
But, like Zahlman points out, its a trick question, and instead of admitting it doesn't know or even prepending "I don't know for sure, but:", it just burps up its best-effort answer. There is no one spirit that represents The Netherlands. If a LLM is so good it "oneshots any question", it should realize it doesn't have a unanimous answer and tell me.