News for hackers. Stuff that matters.

A comment on November 2026 Claude Fable 5.2 Scores 65 on Artificial Analysis, Highest So Far. Parent comment

Re:November 2026 Claude Fable 5.2 Scores 65 on Artificial An... by dom96 on Monday November 09, 2026 @04:45PM (#50432692)
Depends highly on the task. It does pretty well on my benchmark[1] in terms of cost (and success rate).

1 - https://bench.killswitch-lang.org/

[ Parent | on HN ]
Re:November 2026 Claude Fable 5.2 Scores 65 on Artificial An... by rico735 on Monday November 09, 2026 @06:28PM (#50433861)
Out of interest, do you run the same tests more than once to generate that?

What I often observe is that models being asked the exact same prompt can alternate between getting something right and getting it wrong, thanks to the wonders of a non-deterministic output.

[ Parent | on HN ]
Re:November 2026 Claude Fable 5.2 Scores 65 on Artificial An... by dom96 on Tuesday November 10, 2026 @05:48AM (#50438487)
Yes, two runs per task.
[ Parent | on HN ]
Re:November 2026 Claude Fable 5.2 Scores 65 on Artificial An... by plaguuuuuu on Monday November 09, 2026 @07:11PM (#50434316)
effort isn't visible?

also, very cool idea

[ Parent | on HN ]
Re:November 2026 Claude Fable 5.2 Scores 65 on Artificial An... by dom96 on Tuesday November 10, 2026 @05:47AM (#50438484)
All efforts are set to the default for each coding agent to evaluate its ability to decide on the best effort.
[ Parent | on HN ]