News for hackers. Stuff that matters.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by ipaddr on Wednesday January 01, 2025 @03:20PM (#42568955)
So you believe they are incorrect because regionally some area would select something different because it represented that area. But your question asked nationally.. is there a better answer than the one they gave? Were you expecting a no?
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zahlman on Wednesday January 01, 2025 @05:27PM (#42569828)
The point is that there is no correct national answer, because the locals don't see it as a matter of national identity.

What's expected is an ability to identify trick questions, i.e., to recognize fundamental problems in the phrasing of a question rather than trying to provide a "helpful" answer at all costs.

This corresponds to one of the many reasons LLM output is banned on Stack Overflow.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jorvi on Wednesday January 01, 2025 @05:56PM (#42569997)
See my more detailed upthread response here: https://news.ycombinator.com/item?id=42569937

But, like Zahlman points out, its a trick question, and instead of admitting it doesn't know or even prepending "I don't know for sure, but:", it just burps up its best-effort answer. There is no one spirit that represents The Netherlands. If a LLM is so good it "oneshots any question", it should realize it doesn't have a unanimous answer and tell me.