News for hackers. Stuff that matters.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by simonw on Tuesday December 09, 2025 @05:36PM (#46211688)
I've genuinely been answering the question "what if the labs are training on your pelican benchmark" 3-4 times a week for several months at this point. I wrote that piece precisely so I didn't have to copy and paste the same arguments into dozens of different conversations.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Tuesday December 09, 2025 @05:46PM (#46211789)
Oh, no. Does this policing job pay well? /s Seriously: less is more, trust the process, any number of platitudes work here. Who are you defending against? Readers, right? You wrote your thing, defended it with more of the thing. It'll permeate. Or it won't. Does it matter?

You could be done, nothing is making you defend this (sorry) asinine benchmark across the internet. Not trying to (m|y)uck your yum, or whatever.

Remember, I did say linking for convenience is fine. We're belaboring the worst reading in comments. Inconsequential, unnecessary heartburn. Link the blog posts together and call it good enough.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by Barbing on Tuesday December 09, 2025 @07:51PM (#46212725)
Surprised to see snark re: what I thought was a standard practice (linking FAQs, essentially).

I hadn’t seen the post. It was relevant. I just read it. Lucky Ten Thousand can read it next time even though I won’t.

Simon has never seemed annoying so unlike other comments that might worry me (even “Opus made this” even though it’s cool but I’m concerned someone astroturfed), that comment would’ve never raised my eyebrows. He’s also dedicated and I love he devotes his time to a new field like this where it’s great to have attempts at benchmarks, folks cutting through chaff, etc.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Tuesday December 09, 2025 @07:54PM (#46212744)
I'm mainly being snarky about the call for 'same arguments into dozens of different conversations'. The FAQ is a promise to catch them, expect more links. Jump for joy.

Yes, the LLM people will train on this. They will train on absolutely everything [as they have]. The comments/links serve to build engagement more than anything. This is fine, that's not why I'm being critical. The blog-o-sphere motivation, or the belief that this is a common or difficult-to-answer question, perhaps.

I'll readily admit it's silly. That's why I chose snark.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by simonw on Tuesday December 09, 2025 @06:24PM (#46212117)
You don't have to convince me the pelican riding a bicycle SVG benchmark is asinine. That's kind of the point!
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Tuesday December 09, 2025 @06:25PM (#46212127)
Having read the followup post being linked, I'm even more confused. Commenting or, really, anything seems even less worthwhile. That's my point.

You bring the benchmark and anticipated their... cheesing, with a promise to catch them on it. Cool announcement of an announcement. Just do that [or don't]. In a hippy sense, this is no longer yours. It's out there. Like everything else anyone wrote.

Let the LLM people train on your test. Catch them as claimed. Publish again. Huzzah, industry without overtime in the comments. It makes sense/cents to position yourself this way :)

Obviously they're going to train on anything they can get. They did. Mouse, meet cat. Some of us in the house would love it if y'all would keep it down! This is 90s rap beef all over again