Worry not, Pelicans on bicycles had been ranking pretty high on your favorite search engine for a while. I struggle to imagine a world in which it was not already scraped and turned into training data by at least one provider:

1. Models need to be good at the questions we ask them, not the questions we could ask them.

2. The questions, at least partially, are correlated with information people consume.

3. People mostly consume viral content.

4. Ergo you should scrape viral content for training data.

Probably it was added to the training data on the first day when this benchmark was on HN main page. It’s a bad benchmark since then. I don’t know why people still rate it high. Basically, every benchmark becomes pointless after it was published. They are good only to have a picture at the time they’re published first, and not after.

If it was added to the training data on day one, why can't any of the models draw a decent picture of a pelican riding a bicycle?

Most of them were completely unrecognizable back then. Compared to those, this is a huge achievement.

Shouldn't a LLM know how a pelican looks like from biology without being trained on very specific artistic data?

The model likely does know what a pelican looks like; the hard part is translating that into ordered SVG path commands. It's really testing spatial and vector composition, not biological recall.

Imagine you never in your life seen a pelican or any other bird. You only read about them in text. Detailed descriptions but still just descriptions. And now you would need to one shot draw it. Might be a bit hard.

This is for example the result of a taxidermied lion in Sweden when the guy doing the job never ever seen a lion or a photo of them and just worked off descriptions. https://www.snopes.com/articles/344637/the-lion-of-gripsholm...

If it forces LLMs to get better at SVG design, then it's not a terrible incentive.

You can have any SVG you can imagine, as long as it’s a pelican riding a bicycle.