Wow, minimal seems to be the best here interestingly

Really? That very thin neck? I think high is the best, and medium is second best.

I think you have your levels confused - high didn't render.

Incredible. How do you get these out so fast?

I was at my dog's sniffing class for this one, but I had Codex on my Mac (controlled from Codex on my iPhone) run the test prompt with my `llm-meta-ai` plugin, which has the ability to refresh to fetch new models that Meta AI have released: https://github.com/simonw/llm-meta-ai#usage

> dog's sniffing class

There's a blog post in this for sure. And a classic joke.

Sorry I have to ask....um....sniffing class? What is that?

Btw, loved your work with Django!

Is there a problem if I like the minimal more than the others?

I like medium best, but minimal is not bad at all.

Yeah minimal has style

and a fish hat on its head.

It does pretty insane to me that only on max setting (and after waiting 27 seconds) did you actually manage to receive an SVG where the pelican's legs are straddling the bike correctly.

They're not. It's an illusion from the composition.

If you look at the code for each of these (and probably most across models), you can see that the drawing elements are organized by concept. Presumably, this helps accuracy by keeping attention on one concept at a time, but you end up with a stack of conceptualized objects and lose the ability to interleave things.

No doubt these could all do better with a more thorough and specific prompt that coached them away from that technique, but otherwise it's a very subtle detail that a "thinking" process needs to stumble upon independently.

You could use a second prompt to z-order the objects (I tend to break down tasks and pipeline them, rather than trying to stuff everything in a one-shot prompt. Seems to work better for me, could be a holdover from the early days).

They’re not. Note how both pedals are on the same side of the frame. That’s wrong!

Did minimal just hand it off to something else? It seems completely out of line with the rest.

Pelican is the new teapot

birds at minimal and return at max

I like that from xhigh on up its wearing a helmet.

Wearing a helmet is smart, so that checks out.

Not sure this is a good way to benchmark LLM's, no offence, but a single SVG is hardly a measure of a models capability.

[delayed]

>Fortunately, none of the stores caught onto my grandmother's preference for blueberry pie filling.

I think that's exactly it though. When a trend develops for something, the benchmark itself becomes useless both from the social-gaming side and the corporate-trainining-aiming, just general unfairness and dishonesty all around.

and to get around that people have developed personal and private benchmark suites that reflect their own workloads, much like the proposed pie-filling-cost test.

I'd wager a guess that your grandma baked pies.

Any idea why there's so much convergence on the fish in a basket idea?

[delayed]

This is interesting. There's a balance between:

- simple prompt for complex output (bad)

- complex prompt for simple output (bad)

- simple prompt for simple output (good)

- complex prompt for complex output (.... is this good or bad?)

To be fair, people have a genuine preference for images with things facing to the right. And lots of drawings will do 3 motion lines. You'd likely draw a dashed rather than solid separator too, or it would look like a random line. Lots of the AI habits are there because of choices we make.

Exactly, but that doesn't explain the fish.

Leave these poor pelicans and their choice of lunch alone, sheesh.

Doesn't it, or do we not know the answer? I'd ask people to start adding elements to the picture and see how quickly the fish appears.

There aren't that many associations left once we're primed with a bike and a pelican. It's like that old mental trick "What's the colour of the clouds? What's the colour of the snow? What do cows drink?"

> The shorter and vaguer a prompt you give to an LLM, the more it fills in further details using associations they've developed through training.

I can see that for stuff like facing right because characters in images generally do face right, and roads generally do have dashed white line separators. But putting a fish in a basket seems like a stretch of creativity—one that I would rather it not make unless prompted to—so it's surprising that every single model now seems to insist on it.

The models do all seem to converge in ways that, on occasion, wouldn’t seem obvious. But I suppose since they’re all products of the same core training data it’s inevitable that when they’re given the opportunity to fill in the blanks they come to the same conclusions. This seems to be true regardless of what modality you’re working with.

If there isn’t a concerted effort against it the world is going to start to look and feel ever more homogenous.

There's only one average in a set.

Wouldn't that be a sign that those models were trained to optimise that test case specifically, as in, by exhibiting equal or higher performance while using less power?

Indeed. And that they were all trained to give it a red scarf, which was not asked for.

At this point, even if the labs wouldn’t actively pelican-max their models, isn’t the training corpus full of pelicans already? Funny to think how one models choice of adding a red scarf in one of Simon’s earlier experiments might have led to an archetype of pelicans riding bicycles…

Not just pelicans, but pelicans with commentary about how good they are. simonw is accidentally running the world's least efficient hill climbing algorithm.

They all have a red scarf and a fish in a front basket

Fidelity.

Its funny how to the naked eye, changing the effort seems like only a difference on styles

I'm wondering if this is about size. I noticed it took me 2 scrolls on the max, 3 scrolls up the code on whatever in the middle I clicked, and then 4 scrolls up for the minimal.

I'm guessing it's optimizing for "efficient"/less verbose output?

all of them are same enough to clearly be trained on, imo

It feels like an antipattern to add so many reasoning levels as options. How am I as user reasonably going to choose between them?

You use a decider model which reads your prompt and picks the right reasoning level.

But which effort level should I use for the decider model?

(I think there's at least some evidence that it is difficult for decider models to predict the complexity of a task ahead of time with any level of reliability.)

Even now, after so many bicycles, none of them overlaps correctly the elements according to their 3D positions, so it is still very easy to distinguish the drawings done by an LLM from those done by any competent human.

Competence or style?

The first one almost looks like a turkey.