Re:Qwen4-Max
by rhdunn on Monday November 16, 2026 @05:51AM
(#50516182)
Yes, that's another challenge with constructing benchmarks.
You could run the benchmarks using downstream providers like AWS if they are available.
You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.
But ultimately it's another arms race like with viruses and anti-virus software.
You could run the benchmarks using downstream providers like AWS if they are available.
You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.
But ultimately it's another arms race like with viruses and anti-virus software.