News for hackers. Stuff that matters.
Re:MAI-Code-1-Flash by OsrsNeedsf2P on Tuesday June 02, 2026 @02:55PM (#48374585)
So it's trained on the SWE Bench Pro evalset
Re:MAI-Code-1-Flash by lemonish97 on Tuesday June 02, 2026 @02:56PM (#48374609)
What is your evidence for this claim?
Re:MAI-Code-1-Flash by fooker on Tuesday June 02, 2026 @02:59PM (#48374636)
They say hill climbing

https://microsoft.ai/news/building-a-hillclimbing-machine-la...

Unless they specifically clarify that the testing and training benchmarks are completely separate, we have to assume they test on the same 'hill' the model climbs.

Re:MAI-Code-1-Flash by artemisart on Tuesday June 02, 2026 @04:47PM (#48376049)
Hill climbing doesn't mean much but absolutely doesn't imply they cheat on benchmarks. They have more details here https://microsoft.ai/news/introducing-mai-thinking-1/ it seems to be "RL on everything".
Re:MAI-Code-1-Flash by topsycatt on Tuesday June 02, 2026 @04:49PM (#48376065)
That's not accurate. Take a look at the paper to see what it is trained on! And specifically decontamination is called out in A.4

https://microsoft.ai/wp-content/uploads/2026/06/main_2026060...