News for hackers. Stuff that matters.
Re:Qwen4-Max by skohan on Monday November 16, 2026 @08:11AM (#50517251)
Isn't it a bit arbitrary though?

The original concept of this "benchmark" was to see how the model performs on a reasonably complex task that probably isn't in the training data.

Even if they don't fine-tune for this specifically, there are now surely tons of examples of people asking for a pelican on a bike in the training data, and making an SVG isn't that complex of a task for current gen models.

Even with the same model at the same thinking level, you might get different quality out of the same prompt on different runs. If you want to use this as an eval, you would really want to generate 100 of them, and see how often they're anatomically/mechanically accurate.

There are no comments on this comment yet.