News for hackers. Stuff that matters.
Re:GLM-5: From Vibe Coding to Agentic Engineering by RC_ITR on Wednesday February 11, 2026 @04:23PM (#46981235)
The bird not having wings, but all of us calling it a 'solid bird' is one of the most telling examples of the AI expectations gap yet. We even see its own reasoning say it needs 'webbed feet' which are nowhere to be found in the image.

This pattern of considering 90% accuracy (like the level we've seemingly we've stalled out on for the MMLU and AIME) to be 'solved' is really concerning for me.

AGI has to be 100% right 100% of the time to be AGI and we aren't being tough enough on these systems in our evaluations. We're moving on to new and impressive tasks toward some imagined AGI goal without even trying to find out if we can make true Artificial Niche Intelligence.

Re:GLM-5: From Vibe Coding to Agentic Engineering by Rudybega on Wednesday February 11, 2026 @04:50PM (#46981626)
MMLU performance caps out around 90% because there are tons of errors in the actual test set. There's a pretty solid post on it here: https://www.reddit.com/r/LocalLLaMA/comments/163x2wc/philip_...

As far as I can tell for AIME, pretty much every frontier model gets 100% https://llm-stats.com/benchmarks/aime-2025

Re:GLM-5: From Vibe Coding to Agentic Engineering by RC_ITR on Thursday February 12, 2026 @06:44PM (#46996930)
Here's the score for new AIME's, where we know the answers aren't in training.

https://matharena.ai/?view=problem&comp=aime--aime_2026

As for MMLU, is your assertion that these AI labs are not correcting for errors in these exams and then self-reporting scores less than 100%?

As implied by the video, wouldn't it then take 1 intern a week max to fix those errors and allow any AI lab to become the first to consistently 100% the MMLU? I can guarantee Moonshot, DeepSeek, or Alibaba would be all over the opportunity to do just that if it were a real problem.

Re:GLM-5: From Vibe Coding to Agentic Engineering by simonw on Wednesday February 11, 2026 @07:22PM (#46983211)
It has a wing. Look at the code comments in the SVG!
Re:GLM-5: From Vibe Coding to Agentic Engineering by zarzavat on Thursday February 12, 2026 @05:15AM (#46986964)
This test is so far beyond AGI. Try to spit out the SVG for a pelican riding a bicycle. You are only allowed to use a simple text editor. No deleting or moving the text cursor. You have 1 minute.
Re:GLM-5: From Vibe Coding to Agentic Engineering by RC_ITR on Thursday February 12, 2026 @06:49PM (#46996982)
Sorry, is your definition of AGI "doing things worse than humans can do, but way faster?" because that's been true of computers for a long time.
Re:GLM-5: From Vibe Coding to Agentic Engineering by pixl97 on Friday February 13, 2026 @09:14AM (#47002965)
I mean for this particular benchmark, yes.

You'd have to put it in an agentic loop to perform corrections otherwise.

Re:GLM-5: From Vibe Coding to Agentic Engineering by kingstnap on Thursday February 12, 2026 @10:50AM (#46990302)
The benchmarks are harder than you might imagine and contain more wrong answers and terrible questions than you would expect.

You don't need to take my word for it, try playing MMLU yourself.

https://d.erenrich.net/are-you-smarter-than-an-llm/index.htm...

Its not MMLU-Pro btw, which is considerably harder.

Re:GLM-5: From Vibe Coding to Agentic Engineering by RC_ITR on Thursday February 12, 2026 @06:50PM (#46996994)
Sure and AGI will 100% it 100% of the time, even if it is hard.
Re:GLM-5: From Vibe Coding to Agentic Engineering by hieudesu on Saturday February 14, 2026 @08:35AM (#47014426)
Your definition of AGI must be absurd