You are not the first one to think of this. And Simon has thought of it as well, and commented on it several times.
Like... we get it. Simon likes pelicans. If I want to find his boondoggle robo pelican drawings, I know where to find them by now.
No need to spam every single model release thread.
So, I'm curious... What should I be "paying close attention" to?
Here are a bunch of Pelicans I made in my basement. Side by side quant comparisons and something for fun :)
https://stealthgoat.com/fnfl/pelicans3.html https://stealthgoat.com/fnfl/pelicans8.html My favorite, and shows this is not all benchmaxxing, this model is just good at SVG: https://stealthgoat.com/fnfl/pelicans6.html Oh, and, quantization matters: https://stealthgoat.com/fnfl/pelicans2.html
Heres some examples.
I'm curious if an increase in ability to generate this parlays into the model, generally, being better overall or in other adjacent areas?
Thanks again.
The original concept of this "benchmark" was to see how the model performs on a reasonably complex task that probably isn't in the training data.
Even if they don't fine-tune for this specifically, there are now surely tons of examples of people asking for a pelican on a bike in the training data, and making an SVG isn't that complex of a task for current gen models.
Even with the same model at the same thinking level, you might get different quality out of the same prompt on different runs. If you want to use this as an eval, you would really want to generate 100 of them, and see how often they're anatomically/mechanically accurate.
Im not sure how you get the pelican to sit still let alone pedal, but there is a reason ai salaries are so high.
They’re not banned, but must be discouraged. It’s like someone being loud at a museum or using their bright phone at a movie.
If you think it breaks the rules, then flag or downvote. If it just annoys you, collapse it. I'm not sure why this needs to be a whole discussion every time Simon posts a pelican.
If you think HN is transforming into Reddit, I think you may not have visited reddit recently, nor have read the HN guidelines (people have been saying this since 2007) or earlier
> Please don't post comments saying that HN is turning into Reddit. It's a semi-noob illusion, as old as the hills.
Because you keep complaining.
Would love to see an SVG of an Elephant on a skateboard...
“there’s little evidence that AI labs are pelicanmaxxing. Or at least they’re not doing it in a plainly obvious manner. Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict.”
As far as I've seen, among the popular models today, there are still significant gaps in the quality of bike-riding pelikan svgs, making the benchmark still useful. Maybe not as a differentiator of frontier models, but certainly still informative for lesser competitors.
Harder to train off all them.
Aka, let him have his fun, others are making benchmarks too
[^1] https://twitter.com/sunjiao123sun_/status/202455551655137292
"We need to make the pelicans look good, but not so good that it looks like we're using them as an internal benchmark."
2. I think they still are _somewhat_ informative. Like, the same model may produce very different pelicans at different reasoning efforts, even though the training data is the same. So looking at pelicans may give you _some_ information (like "what reasoning effort I want to use that model with?".
https://dylancastillo.co/posts/pelicanmaxxing.html
What I found interesting is that for the most part, the labs really aren't... except if you look at Otter Riding A Plane. That is close to Ethan Mollick's benchmark for diffusion image generation, Otter Using WiFi On A Plane. The results for DeepSeek and GLM look very much like Ethan's typical benchmark image output, which either means they're benchmaxxing... or they're the only two models that understand you ride inside a plane, not on top of it.