News for hackers. Stuff that matters.
Re:Microsoft AI: Seven new world-class models at scalable pr... by spprashant on Wednesday November 18, 2026 @02:10PM (#50550664)
> 98% of the coding quality

Can someone help me understand how they arrive at this figure?

Re:Microsoft AI: Seven new world-class models at scalable pr... by Melatonic on Wednesday November 18, 2026 @04:01PM (#50552068)
Dont worry - it only hallucinates security breaking bugs in 2% of the code it produces. Comparatively :-D

In all seriousness I have no idea how they came to that conclusion. We really need better standards here for evaluating these things but obviously that is difficult.

Re:Microsoft AI: Seven new world-class models at scalable pr... by vibebingo on Wednesday November 18, 2026 @04:16PM (#50552233)
98 can be bad. E.g. uptime or aviation safety.

98 can be good I.e. human test score or funds recovered from debt.

I think for AI tasks it is somewhere between but 2% losses will add up (or you need more loops to correct and hence cost)

Re:Microsoft AI: Seven new world-class models at scalable pr... by WASDx on Wednesday November 18, 2026 @04:33PM (#50552411)
I think they mean "98% of the average score on these old benchmarks which are no longer representative".