Would be great to spend some money (other people's money) on paying some illustrators and artists to do the same for comparison. It could be straight images converted to SVG or direct vector graphics.

Doesn't that somewhat miss the point? The models can't see and check their work, for instance.

I'm not clear what you are saying. I am curious about what humans can do in the same context. It's becoming a standard I think for some tasks. E.g. how does a Gen AI stack up against a human in quality and "cost". Perhaps I'm missing what you are asking.

They can. Ask them to generate SVG , call a renderer and let them inspect the bitmap/png. They can learn about it.

They can, but for this specifc test, they don't. It's a oneshot with no feedback.

Technically you would pay programmers to write SVG code with no visual feedback if you want a straight 1:1 comparison.

Kinda. It's the difference between measuring the end product vs how it's made.

FWIW, I had Astra do it as you suggested: https://pelican-model-gallery.alienfluid.chatgpt.site

I used Light mode and it used up all my limits for the day and had to continue the following day.

This is great! I can see some of the more abstract ones ending up on the wall at SF-MoMA.

Missing GLM-5.3, though.

The GLM 5.1 one where the pelican is missing because it’s broken would be a great candidate. “A pelican enjoying a sunny ride!” is very “this is not a pipe”

You could also have it create a viewer competition selection; show two images without attribution and let the use select which one. Could be cool to see the wisdom of the crowds; could also have price ranges to have subcategories. I bet Astra could do that in a half hour.

Loving the pelican silliness. My ChatGPT is over the moon about it. Gonna miss it when it really is done. Though maybe a pelican riding a bike game could become the benchmark in a year.