News for hackers. Stuff that matters.
Re:Karpathy's Pelican by Wowfunhappy on Sunday August 02, 2026 @06:11PM (#49148943)
> I think it's touching the limit of what one can reasonably expect any intelligent thing to produce with the only direction being "produce an svg of a pelican riding a bicycle".

Oh come now. I am extremely confident that if I hired a professional artist to draw a picture of a pelican riding a bicycle, I would get something inarguably much better than what today's best coding LLMs can produce.

Re:Karpathy's Pelican by skygazer on Sunday August 02, 2026 @06:19PM (#49149011)
At that rate you could use an image model, which was designed for the task. Thats the absurdity of this test. It’s often a text only generation model that has never seen a pelican, coerced into creating an xml graphical representation that a human might recognize. That it does anything passable is already astounding.
Re:Karpathy's Pelican by Wowfunhappy on Sunday August 02, 2026 @06:34PM (#49149116)
I agree, it's hard! That's why it's still a good benchmark.
Re:Karpathy's Pelican by techpression on Sunday August 02, 2026 @06:34PM (#49149118)
It probably has a few million inputs on how a pelican and bicycle looks, not to mention the amount of data on how to create SVG’s. Ask it to create a relaxing spa website and it will, even though it has never seen a spa.
Re:Karpathy's Pelican by didibus on Sunday August 02, 2026 @10:37PM (#49150608)
Most of the models are multi-modal and trained on images no? That's what they claim at least.
Re:Karpathy's Pelican by skygazer on Monday August 03, 2026 @01:42AM (#49151633)
You’re right. Modern frontier models are now multimodal. I used often as weak a hedge, because I know at least his gpt3.5 turbo and llama3.1 generated pelicans were from text only models without image training. The chinese models are interesting, because before their vision models existed they may have been distilling text only models from text output of American vision models, so they could have benefited from the teacher model’s vision capability without being vision models themselves.
Re:Karpathy's Pelican by pj_mukh on Sunday August 02, 2026 @06:55PM (#49149261)
> if I hired a professional artist to draw a picture of a pelican riding a bicycle,

I think AI folks have done a terrible job of communicating this, but replacing a professional simply isn't the point. The point is to serve all the situations where people would've never considered hiring a professional, and where perfection or artistic merit isn't the point (say a personal throwaway recreation of an LOTR world).

And I think in that regard the benchmarks are pretty good.

Re:Karpathy's Pelican by Wowfunhappy on Sunday August 02, 2026 @07:03PM (#49149314)
> replacing a professional simply isn't the point.

I'm not saying it is, just that there's obviously still room for the models to improve on this task.

Re:Karpathy's Pelican by YmiYugy on Sunday August 02, 2026 @08:59PM (#49150039)
IMHO the output is bad enough that I can't imagine a use case for illustrations of this kind.