News for hackers. Stuff that matters.
Re:Qwen4-Max by bee_rider on Monday November 16, 2026 @10:26AM (#50518599)
It’s also worth keeping in mind the other alternative, having internal benchmarks that more accurately reflect your workload, that you know all about, and the LLM provider doesn’t. There doesn’t seem to be a way to maintain such a thing specifically for the purposes of online chatter (both because it isn’t profitable, and because things like “your workload” aren’t possible to define in that case). So we should be happy with what we’ve got.

This is a permanent problem in benchmarking really. Transparency is required for trust but invites over-tuning.

Re:Qwen4-Max by rhdunn on Monday November 16, 2026 @10:42AM (#50518800)
There are tools like promptfoo that can be used for internal benchmarks and evaluations that allow you to compare results across different models and runs. This can be useful for personalized benchmarks and for business applications where you need to evaluate models and/or system prompts.
Re:Qwen4-Max by satvikpendem on Monday November 16, 2026 @12:27PM (#50520175)
Why wouldn't the LLM provider know about your internal benchmarks when they use prompts and chats as training data? They can easily learn about your benchmarks too.
Re:Qwen4-Max by MetalBreaksAndB on Monday November 16, 2026 @01:00PM (#50520636)
Just use ZDR.