News for hackers. Stuff that matters.

A comment on Ask HN: Share your AI prompt that stumps every model. Parent comment

Re:Ask HN: Share your AI prompt that stumps every model by echoangle on Thursday April 24, 2025 @03:13PM (#43786373)
Why not? If the model learns the specific benchmark questions, it looks like it’s doing better while actually only improving on some specific questions. Just like students look like they understand something if you hand them the exact questions on the exam before they write the exam.
[ Parent | on HN ]
Re:Ask HN: Share your AI prompt that stumps every model by namaria on Friday April 25, 2025 @01:17AM (#43790340)
A benchmark that can be gamed cannot be prevented from being gamed by 'security through obscurity'.

Besides this whole line of reasoning is preempted by the mathematical limits to computation and transformers anyway. There's plenty published about that.

Sharing questions that make LLM behave funny is (just) a game without end, there's no need to or point in "hoarding questions".

[ Parent | on HN ]