News for hackers. Stuff that matters.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by Jimmc414 on Tuesday December 09, 2025 @01:57PM (#46209001)
We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. It would be a great way to ensure an initial thumbs up from a prominent reviewer. It's a good benchmark but it seems like it would be a good idea to include an additional random or unannounced similar test to catch any benchmaxxing.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by simonw on Tuesday December 09, 2025 @02:19PM (#46209285)
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by th0ma5 on Tuesday December 09, 2025 @02:27PM (#46209386)
I think constantly replying to everybody with some link which doesn't address their concerns is condescending and disrespectful.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by vanschelven on Tuesday December 09, 2025 @03:40PM (#46210346)
Whatever you think Jimmc414's _concerns_ are (they merely state a possibility) Simon enumerates a number of concerns in the linked article, and then addresses those. So I'm not sure why you think this is so.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by vnvnff on Wednesday December 10, 2025 @05:21AM (#46216124)
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by dugidugout on Tuesday December 09, 2025 @02:39PM (#46209544)
Condescending and disrespectful to whom? Everybody wholsale? This doesnt seem reasonable? Please elaborate.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Tuesday December 09, 2025 @03:52PM (#46210488)
Not sure if I'd use the same descriptions so pointedly, but I can see what they mean.

It's perfectly fine to link for convenience, but it does feel a little disrespectful/SEO-y to not 'continue the conversation'. A summary in the very least, how exactly it pertains. Sell us.

In a sense, saying: "go read this and establish my rhetorical/social position, I'm done here".

It's like if I got to meet an author I really liked, wanted to talk about their work, and they focused more on the catalog than the shared experience.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by tomrod on Tuesday December 09, 2025 @04:33PM (#46210991)
Hell, I would consider myself graced that simonw, yes, THAT simonw, the LLM whisperer, took time out of his busy schedule to send me to a discussion I might have expressed interest in.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Tuesday December 09, 2025 @04:45PM (#46211141)
> send me to a discussion I might have expressed interest in

No, no, remember? Points to the blog you were already reading! Working diligently to build a brand: podcast, paid newsletter, the works.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by tomrod on Tuesday December 09, 2025 @11:12PM (#46214010)
I wasn't speaking to this interaction, and my point is genuine. Simonw has done fantastic work in the LLM space
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Wednesday December 10, 2025 @07:45AM (#46217090)
... and my point remains: he's fine. Could be better. This is one among many interpretations. If he does grace us, he can choose to bait the hook more effectively. Or not.

This interaction is, effectively, a link dropped with an announcement of an announcement. For what has already occurred. Over-fitting, training? You don't say.

If I wanted to be more of an ass, I'd look to argue about hype generation. But I don't, I appreciate any honest effort, which I believe for Simon.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by simonw on Tuesday December 09, 2025 @05:36PM (#46211688)
I've genuinely been answering the question "what if the labs are training on your pelican benchmark" 3-4 times a week for several months at this point. I wrote that piece precisely so I didn't have to copy and paste the same arguments into dozens of different conversations.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Tuesday December 09, 2025 @05:46PM (#46211789)
Oh, no. Does this policing job pay well? /s Seriously: less is more, trust the process, any number of platitudes work here. Who are you defending against? Readers, right? You wrote your thing, defended it with more of the thing. It'll permeate. Or it won't. Does it matter?

You could be done, nothing is making you defend this (sorry) asinine benchmark across the internet. Not trying to (m|y)uck your yum, or whatever.

Remember, I did say linking for convenience is fine. We're belaboring the worst reading in comments. Inconsequential, unnecessary heartburn. Link the blog posts together and call it good enough.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by Barbing on Tuesday December 09, 2025 @07:51PM (#46212725)
Surprised to see snark re: what I thought was a standard practice (linking FAQs, essentially).

I hadn’t seen the post. It was relevant. I just read it. Lucky Ten Thousand can read it next time even though I won’t.

Simon has never seemed annoying so unlike other comments that might worry me (even “Opus made this” even though it’s cool but I’m concerned someone astroturfed), that comment would’ve never raised my eyebrows. He’s also dedicated and I love he devotes his time to a new field like this where it’s great to have attempts at benchmarks, folks cutting through chaff, etc.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Tuesday December 09, 2025 @07:54PM (#46212744)
I'm mainly being snarky about the call for 'same arguments into dozens of different conversations'. The FAQ is a promise to catch them, expect more links. Jump for joy.

Yes, the LLM people will train on this. They will train on absolutely everything [as they have]. The comments/links serve to build engagement more than anything. This is fine, that's not why I'm being critical. The blog-o-sphere motivation, or the belief that this is a common or difficult-to-answer question, perhaps.

I'll readily admit it's silly. That's why I chose snark.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by simonw on Tuesday December 09, 2025 @06:24PM (#46212117)
You don't have to convince me the pelican riding a bicycle SVG benchmark is asinine. That's kind of the point!
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Tuesday December 09, 2025 @06:25PM (#46212127)
Having read the followup post being linked, I'm even more confused. Commenting or, really, anything seems even less worthwhile. That's my point.

You bring the benchmark and anticipated their... cheesing, with a promise to catch them on it. Cool announcement of an announcement. Just do that [or don't]. In a hippy sense, this is no longer yours. It's out there. Like everything else anyone wrote.

Let the LLM people train on your test. Catch them as claimed. Publish again. Huzzah, industry without overtime in the comments. It makes sense/cents to position yourself this way :)

Obviously they're going to train on anything they can get. They did. Mouse, meet cat. Some of us in the house would love it if y'all would keep it down! This is 90s rap beef all over again

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by charcircuit on Tuesday December 09, 2025 @06:47PM (#46212290)
If you want a summary you can have your ai assistant summarize the link.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Tuesday December 09, 2025 @06:49PM (#46212303)
Woooooosh, please see if an LLM can help you. I'm not getting paid for this
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by renewiltord on Wednesday December 10, 2025 @07:11AM (#46216832)
It is SEO-y and I’m sure no small impulse is to drive traffic to his website since he’s primarily an AI influencer.

However, there are always people who are “native” to a platform and field. Pieter Levels is native to Twitter and the nomad community. Swyx is native to Twitter/HN and the devtools community. And simonw is native to at least HN and the LLM-interest community. And various streamers and onlyfans creators do the same with theirs.

Through some degree of releasing things that whatever that community values they build a relationship that allows them greater freedom in participating there. It does create a positive feedback cycle for them (and hopefully the community) that most of them will try to parlay into something else: Levels and the OnlyFans creators are probably best at this monetization of reputation but each of them is doing this. One success step for simonw would be “Creator of Pelican LLM benchmark”.

Once you’ve breached some stable point in the community the norms are somewhat relaxed. But it’s not easy to do that. You have to produce some extraordinary volume of things that people value.

I think, tbh, tptacek here could most effectively monetize if he decided to. But he doesn’t appear to want to so he’s just a participant not an influencer so to speak. Whereas someone like Levels or simonw is both.

It’s just creator economy stuff. Meta discussions like this always pop up. But ultimately simonw is past the threshold of trust. There are people who say “wtf? Why is levels making $50k/mo on a stupid vibe-coded flying game?”

It ain’t the game. It’s the following before the game. The resource is the audience.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by bravetraveler on Wednesday December 10, 2025 @07:26AM (#46216947)
Thanks for posting, I agree. I regret this being taken so pointedly at Simon, just a player in the game.

The best guy spinning the sign puts some effort in, or more crass, the best strippers make you believe.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by dugidugout on Wednesday December 10, 2025 @01:02PM (#46221089)
Well put and thank you for adding much depth here!
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by tptacek on Thursday December 11, 2025 @06:58PM (#46239103)
Wait how do I monetize? Am I leaving money on the table?
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by renewiltord on Friday December 12, 2025 @12:16PM (#46246177)
Not so much leaving money on the table as you have a valuable resource you are underutilizing in this respect. You’d have to cross-cultivate your audience and expand to fill some niche.

That means that instead of dropping facts here you should post it on Substack and link to it here. You’ll be fine with it because you’re trusted here so people will be like “oh it’s fine”.

And then once you have the newsletter going and you have a following there you can do paid placements etc. that don't have to be done by you.

I think you have the highest potential on this front but that doesn't mean there's comparative advantage. It may not be worth it to you. For instance, I would never do it. To keep that wheel spinning you have to keep putting energy into it. And I have other ways of making money, so even with your account it would be a waste.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by tptacek on Friday December 12, 2025 @01:16PM (#46246912)
I'm going to start encrypting the bottom 2/3rds of all my comments and charging a subscription fee for the unlocks. I can be my own Substack, dammit.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by renewiltord on Saturday December 13, 2025 @01:56AM (#46252633)
Hahaha ))
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by tptacek on Saturday December 13, 2025 @02:03AM (#46252661)
U2FsdGVkX1+F16B6AtiHGa1yNbh1HzERm4kYq8UwK4SEbIMUg4N5xZfj
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by th0ma5 on Tuesday December 09, 2025 @02:42PM (#46209596)
No, when did I say that?
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by dugidugout on Tuesday December 09, 2025 @02:56PM (#46209801)
It isn't clear what you said.

You asserted a pattern of conduct on the user simonw:

> I think constantly replying to everybody with some link which doesn't address their concerns

Then claimed that conduct was:

> condescending and disrespectful.

I am asking you to elaborate to whom simonw is condescending and disrespecting. I don't see how it follows.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by armcat on Wednesday December 10, 2025 @07:17AM (#46216885)
Hi Simon! Love your work! Our of curiosity - how many pelican-cycling samples do you produce. Curious about the variance here. Thanks!
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by simonw on Wednesday December 10, 2025 @08:09AM (#46217260)
I've lost count, but there are 85 posts with that tag here: https://simonwillison.net/tags/pelican-riding-a-bicycle/

I need to extract them all into a formal collection.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by karambir on Wednesday December 10, 2025 @08:40AM (#46217593)
I think the parent poster might be asking about generations per model-test. Atleast that's what I understood.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by huxley on Wednesday December 10, 2025 @11:18AM (#46219602)
A coffee-table book? A Natural History of SVG Pelicans
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by jgalt212 on Wednesday December 10, 2025 @08:10AM (#46217271)
Aiden is perhaps misinformed. From a Bing search performed just now.

> Yes, I am familiar with the "pelican riding a bicycle" SVG generation test. It is a benchmark for evaluating the ability of AI models, particularly large language models (LLMs) and multi-modal systems, to generate original, high-quality SVG vector graphics based on a deliberately unusual and complex prompt. The benchmark was popularized by Simon Willison, who selected the prompt because:

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by 100721 on Wednesday December 10, 2025 @10:10AM (#46218542)
Web search-based RAG is very different from having something embedded in a model's training data, though.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by jgalt212 on Wednesday December 10, 2025 @01:59PM (#46221934)
ChatGPT website gives a similar answer. Are they running RAG, or the model?

> Yes — I’m familiar with the “pelican riding a bicycle” SVG generation test.

> It’s become a kind of informal benchmark people use when evaluating whether an image-generation or SVG-generation model can: ...

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by Barbing on Friday December 12, 2025 @12:14PM (#46246164)
Runnin’ confabulations:

>Yes — the “hamster driving a car” prompt is a well-known informal test …

>…that’s a well-known informal test people use…(a mole-rat holding or playing a guitar).

Try any plausible concept. Get sillier and it’s trained to talk about it being nonsense. The output still claims it’s a real test, just a real “nonsense” test.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by th0ma5 on Tuesday December 09, 2025 @02:32PM (#46209451)
If this had any substance then it could be criticized, which is what they're trying to avoid.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by Etheryte on Tuesday December 09, 2025 @04:58PM (#46211306)
How? There's no way for you to verify if they put synthetic data for that into the dataset or not.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by thatwasunusual on Tuesday December 09, 2025 @06:51PM (#46212319)
> We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data.

I may be stupid, but _why_ is this prompt used as a benchmark? I mean, pelicans _can't_ ride a bicycle, so why is it important for "AI" to show that they can (at least visually)?

The "wine glass problem"[0] - and probably others - seems to me to be a lot more relevant...?

[0] https://medium.com/@joe.richardson.iii/the-curious-case-of-t...

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by simonw on Tuesday December 09, 2025 @07:15PM (#46212484)
The fact that pelicans can't ride bicycles is pretty much the point of the benchmark! Asking an LLM to draw something that's physically impossible means it can't just "get it right" - seeing how different models (especially at different sizes) handle the problem is surprisingly interesting.

Honestly though, the benchmark was originally meant to be a stupid joke.

I only started taking it slightly more seriously about six months ago, when I noticed that the quality of the pelican drawings really did correspond quite closely to how generally good the underlying models were.

If a model draws a really good picture of a pelican riding a bicycle there's a solid chance it will be great at all sorts of other things. I wish I could explain why that was!

If you start here and scroll through and look at the progression of pelican on bicycle images it's honestly spooky how well they match the vibes of the models they represent: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-...

So ever since then I've continue to get models to draw pelicans. I certainly wouldn't suggest anyone take serious decisions on model usage based on my stupid benchmark, but it's a fun first-day initial impression thing and it appears to be a useful signal for which models are worth diving into in more detail.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by thatwasunusual on Tuesday December 09, 2025 @09:19PM (#46213279)
> If a model draws a really good picture of a pelican riding a bicycle there's a solid chance it will be great at all sorts of other things.

Why?

If I hired a worker that was really good at drawing pelicans riding a bike, it wouldn't tell me anything about his/her other qualities?!

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by vikramkr on Tuesday December 09, 2025 @11:43PM (#46214165)
The difference is that the worker you hire would be a human being and not a large matrix multiplication that had parameters optimized by a a gradient descent process and embeds concepts in a higher dimensional vector space that results in all sorts of weird things like subliminal learning (https://alignment.anthropic.com/2025/subliminal-learning/).

It's not a human intelligence - it's a totally different thing, so why would the same test that you use to evaluate human abilities apply here?

Also more directly the "all sorts of other things" we want llms to be good at often involve writing code/spatial reasoning/world understanding which creating an svg of a pelican riding a bicycle very very directly evaluates so it's not even that surprising?

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by simonw on Tuesday December 09, 2025 @10:10PM (#46213561)
I wish I knew why. I didn't think it would be a useful indicator of model skills at all when I started doing it, but over time the pattern has held that performance on pelican riding a bicycle is a good indicator of performance on other tasks.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by jtbaker on Tuesday December 09, 2025 @09:53PM (#46213477)
a posteriori knowledge. the pelican isn't the point, it's just amusing. the point is that Simon has seen a correlation between this skill and and the model's general capabilities.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by suspended_state on Wednesday December 10, 2025 @02:22AM (#46215042)
Your comment is funny, but please note: it's not drawing a pelican riding a bike, it's describing in SVG a pelican riding a bike. Your candidate would at least displays some knowledge of the SVG specs.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by falcor84 on Wednesday December 10, 2025 @05:22AM (#46216127)
For better or worse, a lot of job interviews actually do use contrived questions like this, such as the infamous "how many golf balls can you fit in a 747?"
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by theshrike79 on Wednesday December 10, 2025 @06:48AM (#46216683)
What if the employee can draw a bike and a pelican, but not a pelican on a bike?
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by wisty on Tuesday December 09, 2025 @07:18PM (#46212511)
It's not nessessarily the best benchmark, it's a popular one, probably because it's funny.

Yes it's like the wine glass thing.

Also it's kind of got depth. Does it draw the pelican and the bicycle? Can the penguin reach the peddles? How?

I can imagine a really good AI finding a funny or creative or realistic way for the penguin to reach the peddles.

An slightly worse AI will do an OK job, maybe just making the bike small or the legs too long.

An OK AI will draw a penguin on top of a bicycle and just call it a day.

It's not as binary as the wine glass example.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by thatwasunusual on Tuesday December 09, 2025 @09:16PM (#46213266)
> It's not nessessarily the best benchmark, it's a popular one, probably because it's funny.

> Yes it's like the wine glass thing.

No, it's not!

That's part of my point; the wine glass scenario is a _realistic_ scenario. The pelican riding a bike is not. It's a _huge_ difference. Why should we measure intelligence (...) in regards to something that is realistic and something that is unrealistic?

I just don't get it.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by vikramkr on Tuesday December 09, 2025 @11:44PM (#46214172)
If the thing we're measuring is a the ability to write code, visually reason, and handle extrapolating to out of sample prompts, then why shouldn't we evaluate it by asking it to write code to generate a strange image that it wouldn't have seen in its training data?
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by Fnoord on Wednesday December 10, 2025 @12:50AM (#46214531)
> the wine glass scenario is a _realistic_ scenario

It is unrealistic because if you go to a restaurant, you don't get served a glass like that. It is frowned upon (alcohol is a drug, after all) and impractical (wine stains are annoying) to fill a glass of wine as such.

A pelican riding a bike, on the other hand, is realistic in a scenario because of TV for children. Example from 1950's animation/comic involving a pelican [1].

[1] https://en.wikipedia.org/wiki/The_Adventures_of_Paddy_the_Pe...

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by mzl on Wednesday December 10, 2025 @07:45AM (#46217091)
A better reason why wine glasses are not filled like that is that wine glasses are designed to capture the aroma of the wine.

Since people look at a glass of wine and judge how much "value" they got based partly on how much wine it looks like, many bars and restaurants choose bad wine-glasses (for the purpose of enjoying wine) that are smalle and thus can be fulled more.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by theshrike79 on Wednesday December 10, 2025 @06:48AM (#46216675)
It's just a variant of the wine glass - something that doesn't exist in the source material as-is. I have a few of my own I don't share publicly.

Basically in my niche I _know_ there are no original pictures of specific situations and my prompts test whether the LLM is "creative" enough to combine multiple sources into one that matches my prompt.

I think of if like this: there are three things I want in the picture (more actually, but for the example assume 3). All three are really far from each other in relevance, in the very corner of an equilateral triangle (in the vector space of the LLM's "brain"). What I'm asking it to do is in the middle of all three things.

Every model so far tends to veer towards one or two of the points more than others because it can't figure out how to combine them all into one properly.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by 0cf8612b2e1e on Tuesday December 09, 2025 @07:26PM (#46212565)
I assume all of the models also have variations on, “how many ‘r’s in strawberry”.
Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by theshrike79 on Wednesday December 10, 2025 @06:42AM (#46216646)
The easiest way to fix these is give the model an environment to run code.

Any model can easily one-shot a python script that can count the occurrence of any letter anywhere and return the result.

It's just a tooling issue. You really can't "train" an LLM to do it because tokenisation and ... stuff.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by 0cf8612b2e1e on Wednesday December 10, 2025 @11:38AM (#46219889)
I am not convinced they are executing code. Otherwise I would expect LLMs to not frequently guess the result of math questions.

Of course you could train it. Some quick scripting to find all words with repeat letters, build up sample sentences (aardvark has three a,) and you have hard coded the answer to simple questions that make your LLM look stupid.

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by theshrike79 on Wednesday December 10, 2025 @02:52PM (#46222770)
I have personally observed Grok running Python code in a chat to determine the current date so it could accurately tell me whether the 20th is a Friday (it wasn't in that specific month)

.. it did that in a story prompt that didn't happen in a) our world b) the current time =)

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by Workaccount2 on Tuesday December 09, 2025 @07:28PM (#46212581)
It would be easy to out models that train on the bike pelican, because they would probably suck at the kayaking bumblebee.

So far though, the models good at bike pelican are also good at kayak bumblebee, or whatever other strange combo you can come up with.

So if they are trying to benchmaxx by making SVG generation stronger, that's not really a miss, is it?

Re:Mistral Releases Devstral 2 (72.2% SWE-Bench Verified) an... by majormajor on Tuesday December 09, 2025 @08:29PM (#46212998)
That depends on if "SVG generation" is a particularly useful LLM/coding model skill outside of benchmarking. I.e., if they make that stronger with some params that otherwise may have been used for "rust type system awareness" or somesuch, it might be a net loss outside of the benchmarks.