News for hackers. Stuff that matters.
Re:Claude Opus 4.6 by joshuahedlund on Thursday February 05, 2026 @01:30PM (#46903012)
Any ideas why verified has stagnated? It was increasing rapidly and then basically stopped.
Re:Claude Opus 4.6 by Snuggly73 on Thursday February 05, 2026 @01:53PM (#46903367)
it has been pretty much a benchmark for memorization for a while. there is a paper on the subject somewhere.

swe bench pro public is newer, but its not live, so it will get slowly memorized as well. the private dataset is more interesting, as are the results there:

https://scale.com/leaderboard/swe_bench_pro_private