News for hackers. Stuff that matters.
Re:System Card: Claude Mythos Preview [pdf] by pants2 on Tuesday April 07, 2026 @02:37PM (#47679508)
We're gonna need some new benchmarks...

ARC-AGI-3 might be the only remaining benchmark below 50%

Re:System Card: Claude Mythos Preview [pdf] by Leynos on Tuesday April 07, 2026 @04:08PM (#47680706)
Opus 4.6 currently leads the remote labor index at 4.17. GPT-5.4 isn't measured on that one though: https://www.remotelabor.ai/

GPT 5.4 Pro leads Frontier Maths Tier 4 at 35%: https://epoch.ai/benchmarks/frontiermath-tier-4/

[dead]
Re:System Card: Claude Mythos Preview [pdf] by saberience on Wednesday April 08, 2026 @06:14AM (#47688042)
I've never understood the point of things like HLE, it doesn't really prove or show anything since 99.99% of humans can't do a single question on this exam.

That is, it's easy to make benchmarks which humans are bad at, humans are really bad at many things.

Divide 123094382345234523452345111 by 0.1234243131324, guess what, humans would find that hard, computers easy. But it doesn't mean much.

Humanity's last exam (HLE) couldn't be completed by most of humanity, the vast majority, so it doesn't really capture anything about humanity or mean much if a computer can do it.

[dead]
Re:System Card: Claude Mythos Preview [pdf] by elfly on Wednesday April 08, 2026 @03:11PM (#47694851)
I mean they could just feed the solutions into the training data. Then suddenly the bot will do real good at HLE.
Re:System Card: Claude Mythos Preview [pdf] by mbesto on Thursday April 09, 2026 @02:24PM (#47707507)
Exactly. This is called overfitting and it's most definitely a thing.
Re:System Card: Claude Mythos Preview [pdf] by mbesto on Thursday April 09, 2026 @02:23PM (#47707500)
> We're gonna need some new benchmarks...

You can't consistently benchmark something that is qualitative by nature. I'm struggling to understand how people don't understand this.