News for hackers. Stuff that matters.
Re:Benchmarks in Leipzig by jll29 on Saturday June 06, 2026 @05:55PM (#48429390)
Can anyone comment on the "distance to publication-worthiness" of the typical question from this set?
Re:Benchmarks in Leipzig by christianstump on Sunday June 07, 2026 @04:09AM (#48432866)
Infinitly far away. These questions are about "have you understood and can you apply existing research" not about "create new research".

For humans, these two correlate quite strongly. So we ask PhD students to work on the former to prepare and to become better at the latter.

For LLMs, it remains unclear if there is any correlation.