> If it's on HN and is a meme at this point, it will end up in the training set.

> It's kind of fun to imagine that there is an intern in every AI company furiously trying to get nice looking svg pelicans on bicycles.

Google listened and assigned a few interns, I guess, for the task

Figure of speech.

>Why not just generate svgs with any editing software?

How do you do that from the whiteboard/notebook?

Everyone says that until ms, apple, google, etc introduces this polished feature as part of their tablet imaging editing/document collaboration office suite. Then everyone can;t get enough of it.

> It's a small task whose result is an input to something else - ideal for a tool for LLM (or script)

Have you actually tried this? If it's a small task then why can't LLMs actually do it? Kind of like the pelican test. Every LLM can do a SVG pelican but most of the results are unusable in practice for production/productivity purposes.

Every LLm can extract some half-broken SVG hallucination of your jots, but I'm asking for an app that's polished and works accurately 99% of the time, which means a dedicated app/model tuned specifically for this task, and nobody makes this yet.

I don't feel like I'm a representative sample of general userbase, so I won't dispute this too hard. For me, it's rare but it does happen. Still, 90% of my mobile experience is "how is this so broken, why is it missing the obvious follow-up feature that completes a workflow, and why, again, am I not using a laptop for this?".

> If it's a small task then why can't LLMs actually do it?

What if they actually can?

This is exactly what Karpathy was talking about the other day[0]. If your only LLM experience is the default chat app in chat mode, then this doesn't make sense. If your experience includes using agentic harness like Claude Code/Cowork / Codex/ChatGPT Work, then it's obvious.

The LLM app in "chat mode" will just zero-shot you a pelican, or digitize your notes into SVG, and maybe improve on it a bit if you complain about the result. The same app[1] in "agentic mode" will digitize your notes, refine the SVG against the original image, reduce it to probable strokes based on high-frequency patterns in the photo and knowledge of how people write, OCR it, save both the text and SVG to your library, then extract parts you wanted and insert them into a document, typeset the rest of it, and print it on your printer - all in response to a single prompt.

LLMs in agentic mode are very good at stringing long and advanced workflows; today, if you give them shell access or computer use, they will use regular apps for you if that's the easiest way to accomplish the goal.

> Have you actually tried this?

Yes![2].

That's why I can confidently say: there is no limit to this pattern. LLMs can handle apps just fine; giving them tools just saves time and tokens.

Another example: I'm currently resurrecting a 30 year old game, and the other day I asked Claude to hack on a way to run it headlessly, so I can use my computer while it plays with it. It did, and then immediately leveraged it to just causally run several instances of the game in parallel, entirely in the background, whenever it needed to playtest something.

--

[0] - https://news.ycombinator.com/item?id=50047422 - was on the frontpage two days ago.

[1] - Claude and ChatGPT now have both modes in their mobile, web and desktop apps. I personally use Claude Code a lot, but exclusively through these surfaces (I can't be arsed to deal with TUI at home anymore). IMO, these tools are already within reach of regular users - the users just didn't notice it yet.

[2] - Not with SVGs, but equivalent with voice recordings, and yes, Claude Code set up and is managing the whole local processing pipeline. Recordings go in, Markdown (or more advanced signal processing) comes out, and in itself, it's not a task, but merely a step.

One fictional example of what I mean: lets say I draw someone holding a purse. That would be a shape of their torso, with the purse overlaid. I could move those independently, and moving the purse would show the torso underneath.

However, using a png to svg, it would often just outline everything. So it's not a torso shape with a purse in front. It's a purse with a torso cut out around it, maybe even merged together as lots of individual shapes. If I were to delete the shapes making up the purse, it could be quite random what was left. Maybe some of the shapes did a double duty. Maybe it's a thousand small squares to make the purse. Maybe there is just a hole behind the purse etc.

So in the latter case, the svg isn't made up of logical elements, it's just made up of shape after shape after shape to mimic the pixels. But it's not usable as vector.

> How many pelican riding bicycle SVGs were there before this test existed? What if the training data is being polluted with all these wonky results...

But Simon has demonstrated before that asking for any animal doing something silly will still get similar results from the same model.

> That's the result of training on the same datasets and scraped web contaminated with other models' outputs

Maybe I'm misinterpreting though, or just semantics. I would consider contamination, or whatever the first example was that they all matched, as specific training data, along with all the duplications of. I doubt it's model collapse to the same pelican, without the same original example.

Expect Google models to be benchmaxxed as much as possible, but don't expect commensurate performance on real-world tasks.

This is not true. It is needless negative hyperbole that IMHO doesn't belong on HN.

If what you say is true, then the next set of benchmarks will be written to illuminate the true reflection of real-world tasks. Then those get benchmaxxed so managers can show "numbers go up."

Repeat enough times, and you get materially and genuinely improved products.

The burden is on the press, outside researchers, and general tech industry to both write and care about the metrics that matter.

In large companies (Fortune 1000 etc.), you don't have much of a choice, the culture was eroded long ago, and corporate governance by metrics is a lesser of evils. But thankfully, not all companies are there yet.

They look remarkably good, sure -- but not perfect.

I note that even these bikes are not actually rideable. For a bike to be stable, the front wheel axle needs to extend forward a little ahead of the fork - the fork must ”bend forwards”. This is what actually keeps the bike upright, not gyroscopic effects or some such. Given LLMs are generally good at basic maths, im surprised they get this wrong.

Maybe a bit less so in this case though. As per the announcements, the model was actually trained for this prompt, so was fed some training data. I can well imagine the data there was just not so good.

People drawing bikes, scroll down to see they having made renders of all the drawings. Hilarious

You can also ride a negative rake fork. They are less stable, yes, but a far cry from not actually rideable, cf https://www.instagram.com/yourforkisonbackwards/

I'll also note that, using a piece of paper as a straight edge against my screen, there is a very slight forward curve on the low and a bit more on the high responses. Medium does appear to be perfectly straight.

But even with zero offset, there is another thing in play. The steering axis is at an angle, a fork never points straight down. So even with zero offset, you still get a trail, which is the difference between the contact point of the wheel at the ground, and where the steering axis meets the ground. This trail is really what matters in the end, not how you get there.

See https://en.wikipedia.org/wiki/Bicycle_and_motorcycle_geometr...

But even better, play with (ctrl+f) "trail" in this simulation: https://ciechanow.ski/bicycle/

There you see the point. Even with zero offset, there is still a trail.

https://www.walmart.com/ip/Huffy-26-inch-Rock-Creek-Men-s-Mo...

(apologies if I'm misunderstanding).

Whether having zero offset or not isn't what matter in the end, but the trail. And you can achieve that with little offset as long as the fork is angled.

(so the grandparent is a bit right in their critique, but not for the reason stated)

You're nit-picking on a detail no one actually cares about.

1. Modeling and analyzing spinning, three-dimensional objects goes far beyond "basic maths".

2. The prompt was for an SVG of a realistically impossible situation (pelican riding a bicycle), which together already implies a stylized depiction. Why on Earth would you expect it to generate a 3D model of a bike, do a physics simulation on it, iterate until it satisfies certain contraints which are only hinted at in the prompt, and then project it down to a simplified 2D cartoon drawing? That's not how LLMs work, and that's not how we want them to work.

Subtle highlights on the metal frame, sweat beads, jaunty little hat…it has style. At least compared to the rather literal ChatGPT outputs. https://static.simonwillison.net/static/2026/gpt-pelicans-gr...

Is the prompt context telling the models what their effort level is, and the ambient presence of that concept of ‘high effort’ in the context window is leaking through and affecting the portrayal?

https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/chat_templ...

xhigh: "Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer."

low: "Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration."

Argon: https://jonclegg.github.io/pacman-bakeoff/dev/#gemini-4-argo...

Opus 5.5 https://jonclegg.github.io/pacman-bakeoff/dev/#claude-opus-5...

Also, that first tweet contains a commentary that, for me, explain a lot about the public misconception about how current AI works:

> wait, does this mean you guys are actively training svg generation? i was mostly amazed by it thinking the models doing better at it by itself due to becoming better at reasoning.

No, it's not better reasoning that drives current AI, it's better examples we have been giving them. Whole army of people generating content for the frontier labs.

Also, reasoning improving the out of distribution part, like computer use, tool use etc.

Another case where allowing oneself to anthropomorphise LLMs a little yields better intuitions.

People don't get more skilled by raw reasoning, either. They improve their skills with better textbooks (pre-training data), more practice exercises (RL data), and generally goofing around (self-play).

https://en.wikipedia.org/wiki/Pelican#/media/File:Pelikan_Wa...

It's a simple manner of prompt...generate concept art with nanobanana, measure, llm creates svg with text, measure the distance between rasterized svg embedding to nanobanana art....do post-training RL on a 3 million such images and matching embeddings.

I swear this will be a canonical example of Goodhardt's Law in action.

You have to switch to a new benchmark that isn't so easy to run a verifiable rewards pipeline on...Perhaps a longer horizon task where it's easy to mess up midway.

Yes, Simon has made pelicans a benchmark and now models have grown very good at rendering them. Is there a reason to believe that this isn’t manifesting as useful skill in general in SVG generation?

It would perhaps be slightly interesting if the prompt was run 100 times for each model. I doubt the pelicans would look identical.

However, how the front fork rake affect stability is I believe a little complex. I doubt this old article is the last word on the subject, but it's a great read nonetheless, and note especially the ultra-stability of the reversed fork bike!

https://physicstoday.aip.org/features/from-the-archives-the-...

I've noticed that chatgpt is working harder. But still I prefer Claude, it just feels nicer to me.

As one point, the high pelican's arm is see-through/messed up.

They do look worlds better than previously, to be fair.

Modern models can make nearly photorealistic games in threejs. So why do they struggle with these pelicans?

However, result quality really drops off once you get out of the top 5 or so.

(NSFW note: all buttocks are covered by fairly conservative bikini bottoms and show no genitals, bulges, etc.)

The test is always going to be hard though, because it's an absurd image to ask for.

UPDATE: I like this theory about using an agentic harness that can render and iterate better: https://news.ycombinator.com/item?id=50073879#50074395

this is a really profound insight

https://www.gianlucagimini.it/portfolio-item/velocipedia/

HN discussion: https://news.ycombinator.com/item?id=17699017

https://www.wired.com/2016/04/can-draw-bikes-memory-definite...

The pretty 3d stuff you see are agents controlling a Blender MCP in a loop where they look at the scene, think, adjust the scene, repeat.

(As I said in a separate comment above, results are all bikini-clad and not revealing. However, whether it is NSFW is up to you/your work policies.)

Are you sure they're not fetching free asset packs for that?

I even built an environment that allows execution of Python code to render with Blender but allows no other interactions (including networking, I sandbox that off). They can create good 3D models just fine - here's a Faberge Egg by Gemini 4 Argon: https://tools.simonwillison.net/blender-viewer?url=https%3A%...

Thanks for doing all this research!

Here's 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars' for Gemini 4 Argon: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

Some other models for comparison: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

1. the pedestrian traffic light's hand and "wait, citizen" text are not inside the light itself

2. The armadillo's nose is outside of the astronaut helmet

Isn't that just proof that the pelican has been benchmaxxed?

wasn't the whole point of pelicans is that labs *wouldn't* train for that?

so now the benchmark is ruined

Give me reverse engineering bench and agentic coding.

Leave making visuals to humans who actually have aesthetic sense unlike whatever made this mess

<https://tools.simonwillison.net/svg-render#%3Csvg%20font-fam...> <https://tools.simonwillison.net/svg-render#%3Csvg%20font-fam...>

- I was having trouble embedding the SVGs as links, and ChatGPT recommended Simon Willison's SVG Renderer as a workaround.

Just my 2 cents on the pelicans benchmark: I think it might help if you were a bit more explicit about not taking this one benchmark too seriously. I know you say that at times, but at other times (such as the comment I'm replying to) you elide it which can read as a seriousness about the benchmark.

Medium and high are fixies apparently, (or they have a worm gear, although the hub doesn't look big enough for that), since neither has a derailleur. That would also explain the lack of brakes at least.

Low is still obviously failing the "test". It's pedals are not actually attached to the bike either.

Medium has a handlebar that goes through the pelican's beak. So I'd say also failing.

For the high, the only nitpick would be that the bell isn't quite on the handlebars and the stripes on his scarf don't line up correctly.

This is the sheet: https://fernandoipar.com/content/music/poly-d/gemini-argon/c... (the annotations on it are by me)

And this is what Argon created: https://fernandoipar.com/content/music/poly-d/gemini-argon/b...

For this simple prompt: https://fernandoipar.com/content/music/poly-d/gemini-argon/p...

I'm honestly impressed.