News for hackers. Stuff that matters.

A comment on GLM-5: From Vibe Coding to Agentic Engineering. Parent comment

Re:GLM-5: From Vibe Coding to Agentic Engineering by Rudybega on Wednesday February 11, 2026 @04:50PM (#46981626)
MMLU performance caps out around 90% because there are tons of errors in the actual test set. There's a pretty solid post on it here: https://www.reddit.com/r/LocalLLaMA/comments/163x2wc/philip_...

As far as I can tell for AIME, pretty much every frontier model gets 100% https://llm-stats.com/benchmarks/aime-2025

[ Parent | on HN ]
Re:GLM-5: From Vibe Coding to Agentic Engineering by RC_ITR on Thursday February 12, 2026 @06:44PM (#46996930)
Here's the score for new AIME's, where we know the answers aren't in training.

https://matharena.ai/?view=problem&comp=aime--aime_2026

As for MMLU, is your assertion that these AI labs are not correcting for errors in these exams and then self-reporting scores less than 100%?

As implied by the video, wouldn't it then take 1 intern a week max to fix those errors and allow any AI lab to become the first to consistently 100% the MMLU? I can guarantee Moonshot, DeepSeek, or Alibaba would be all over the opportunity to do just that if it were a real problem.

[ Parent | on HN ]